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ABSTRACT 



This thesis describes an experiment conducted at the 
Nsv'al Postgraduate School (NFS) during the period 30 October 
1984 through 30 November- 1984. Sp e c i f i c a 1 1 y > the experiment 
compares the use of continuous speech recognition equipment; 
discrete speech recognition equipment; and keyboard to input 
commands in 3 command and control environment. This was 
accomplished by using the Naval Warfare Interactive 
Simulation System (NWISS) as a vehicle to pose military 
problems to subjects in a variety of light and noise 
e n v i r omnents. 

Although the results are not conclusive; they do show a 
Definite advantage in using continuous speech or keyboard 
entry modes over discrete speech modes. Continuous speech 
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table of contents 

I INTRODUCTION 

A. OVERVIEW 

B. PURPOSE OF THE THESIS 

C. SUMMARY 

I! SPEECH TECHNOLOGY 

A. INTRODUCTION 

E. DEFINITION OF TERMS 

C. PROBLEMS OF SPEECH RECOGNITION 

D. OVERVIEW OF SPEECH RECOGNITION TECHNOLOGIES — 

1. General 

a. Sound Analysis 

c. C 1 a ss i f i ca t i on 

2. Discrete Speech Recognition 

3. Continuous Speech Recognition 

E. PREVIOUS EXPERIMENTS CONDUCTED 

F. DESCRIPTION OF HARDWARE UTILIZED 

1. Discrete Speech Recognition Equipment 

2. Continuous Speech Recognition Equipment — 

3. Laboratory Equipment 

G. SUMMARY 

III. THE NAVAL WARFARE INTERACTIVE SIMULATION SYSTEM — 

A. INTRODUCTION 

B. WHY NW I SS WAS USED 



10 

10 

1 2 

1 4 

15 

15 

i c . 

IS 

19 

19 

19 

1 9 

21 

24 

28 

2S 

29 

31 

31 

33 

'“V'O 

w'O 

no 

O 



5 



C SCENARIOS 35 

1. Scenario A 36 

2. Scenario B 36 

3. Scenario C 3S 

4. Scenario D 3S 

D. MEASURES OF EFFECTIVENESS 42 

E. DATA COLLECTION 45 

F. SUMMARY 46 

VOICE EXPERIMENT BACKGROUND 47 

A. INTRODUCTION 47 

B. BACKGROUND OF SUBJECTS 47 

C. ENVIRONMENTAL CONDITIONS 49 

D. SCHEDULING OF SESSIONS 50 

E. SCENARIO/ENVIRONMENT SEQUENCE 52 

F. CONSTRAINTS 53 

0. SUMMARY 54 

CONDUCT OF THE EXPERIMENT 56 

A. INTRODUCTION 56 

B. LABORATORY CONFIGURATION AND CONTROL 56 

1. Configuration 56 

2. Control of the Lab 5S 

C. TRAINING 59 

1. General 59 

2. Environment During Training 59 

3. Discrete Voice Training 60 

4. Continuous Voice Training 6i 



6 



5. Retraining 62 

D. PRACTICE SESSIONS 64 

1. Initial Individual Tasks 64 

2. NWISS Practice Sessions 66 

a. General 66 

b . Purpose 66 

c. Preparation for the Practice Session - 6S 

d Conduct of the Practice Session 6S 

e. Completion of the Practice Session 69 

E . TEST SESSIONS 69 

F. RESCHEDULING 70 

S. SUMMARY 71 

DATA ANALYSIS 72 

A. INTRODUCTION 72 



DATA SUMMARY 

STATISTICAL METHODS 



D. ANALYSIS SC 

1. Best Environment for Each Input Method SO 

2. Best Input Method for Each Environment Si 

3. Best Overall Input Method S2 

A. Learning Curves S3 

E. SUMMARY 83 

UII CONCLUSIONS AND RECOMMENDATIONS 85 

A. INTRODUCTION Q5 

B. CONCLUSIONS S5 

C. IMPLICATIONS FOR DATA ENTRY METHODS 88 



7 



D RECOMMENDATIONS 90 

E SUMMARY 91 

APPENDIX A VOCABULARY LISTS FOR EXPERIMENT 92 

APPENDIX B. SCENARIO BRIEFINGS 107 

APPENDIX C. VOICE EXPERIMENT TEAM MEMBERSHIP 112 

APPENDIX D. ADMINISTRATIVE BRIEFING 113 

APPENDIX E. SAMPLE TASKS TO BE PERFORMED 114 

APPENDIX F. INDIVIDUAL TEST SESSION DATA 115 

LIST OF REFERENCES 117 

INITIAL DISTRIBUTION LIST 120 



b 



ACKNOWLEDGEMENTS 



■? n c q 




3 5 5 



We bish to gratefully acknowledge the guidance and 
yragsmsnt of our thesis advisor CDR Joseph Stewart, 
out whose time and assistance this thesis would not have 
possible. We would also like to acknowledge the 
'-rise ana assistance provided by our second reader Gary 
Foock and by Jay Martin. Additionally# our thanks to the 
ants of the Naval Postgraduate School who participated 
ub sects in our research. 



I. INTRODUCTION 



A OVERVIEW 

hany offices in the public and private sectors of the 
United States today are concerned with increasing the 
efficiency and productivity of the work force. Modern 
teen no logy is often involved in accomplishing these ends. 
Providing some advanced technology to the worker may, 
however, create a situation which causes the job to be more 
complex than it was originally, and many people are not 
prepared for the new environment. This is a very common 
problem in the use of computers. There has been a push to 
simplify this man -machine interface, and many input and 
output methods have been explored. With the increasing 
emphasis on man-machine interface and communication, 
automatic speech recognition (ASR) is expected to become a 
major force. ASR involves communicating with a machine. When 
the ASR equipment correctly recognizes what is being spoken, 
it sends out a predetermined ASCII string of characters to a 
computer, or it could be used to send a signal to turn off 
lights or turn on power switches. The advantages of using 
speech in communicating with a machine include its 
convenience, speed, universality, and the possibility of 
avoiding or reducing the training of workers through its 
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Automatic speech recognition is considered to be the 



most difficult and complex problem in the field of voice 
processing It uses large-scale integration (LSI) and very 
large-scale integration (VLSI) chip design/ signal 
processing; acoustic phonetics; natural language theory; 
linguistics; mathematics of stochastic processes; and 
computer science techniques [Ref. 13. 

Speech, in its natural form/ is the fastest and .most 
convenient means for humans to communicate with each other. 
While humans may be hindered by a language barrier; machines 
can be trained to learn and respond to any language. 
Adtiitionailyi ASR is convenient in that hands and eyes are 
free and the speaker has mobility; which other 
communications modes do not offer. With a microphone 
headset; the speaker is free to move about and perform 
concurrent functions such as viewing graphics screens or 
other decision aids in order to make decisions,- or 
interfacing with other people. 

Progress toward economically feasible voice recognition 
systems has been slow; but ASR products are gradually 
Decoming available; primarily for industrial applications. 
Industrial applications include “hands — busy 11 activities 
involving inspections usually associated with quality 
control; flight training simulators for entering commands; 
baggage/parcel sorting; voice control of machine tooling; 
and home appliance control. The United States National 



Security Agency (NBA) has developed speech recognition 
algorithms allowing it to spot key words in intercepted 
verbal transmissions from unfriendly nations. CRef. 1] 
ti i 1 i t a r y applications include aircraft cockpit voice-entered 
commands and computer-assisted training of air traffic 
controllers. 

B PURPOSE OF THE THESIS 

In the rush to incorporate emerging technology in 
various fields, there have been gaps in the research to show 
whether or not ASR enhances productivity and efficiency. A 
comprehensive comparison of computer input technologies in a 
military command and control environment has not been 
performed. This thesis describes an experiment conducted at 
the Naval Postgraduate School (NPS) during the period 30 
October 1984 to 30 November 1984. This experiment was the 
first attempt to compare two different voice recognition 
systems of differing technologies and keyboard (typing) 
modes of input in issuing commands in a controlled wargame. 

The principle objective of the thesis was to design and 
implement an experiment in which the three modes of input 
could be utilized/ resulting in data that would be used to 
determine a measure of performance to make a direct 
comparison of the devices. The authors became involved in 
the project during the conceptual stage/ and performed the 
design/ setup/ conduct/ and analysis of the experiment. The 
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goal of the experiment was to develop wargame scenarios that 
would simulate military conflict situations requiring 
decision making and issuance of orders under time 
constraints. There had to be sufficient replications of 
trials; control of procedures; and limitation of variables 
to allow drawing s t a t i s t i c a 1 1 y significant conclusions. 

Numerous experiments concerning automatic speech 
recognition have Deen conducted at NPS. This experiment was 
conducted in the Command; Control; and Communications (C3) 
Laboratory; also known as the War gaming Analysis and 
Research (WAR) Laboratory; using a VAX 11/780 computer built 
by Digital Equipment Corporation. The Naval Warfare 
Interactive Simulation System (NWISS)# a computer based 
'Naval wargame; was used to simulate military command and 
control situations for students serving as subjects for the 
experiment. The two types of speech recognition equipment 
available in the C3 Lab were the Vertex 3000 continuous 
speech input system and the Threshold T600 discrete speech 
system. A Digital VT102 terminal; including a video screen 
and keyboard; was used for the typing mode of input to the 
computer. A group of students was assigned to each of the 
three input devices. Different scenarios and environments 
were designed for the experiment; but all three command 
input methods were used for each combination of scenario and 
environment in order to determine which input mode was best 
in different situations. 
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C SUMMARY 

This thesis provides a description of the experiment 
that was conducted; including background material and 
procedures leading up to the experiment. Basic speech 
recognition technology i s described; and a review of past 
experiments with speech recognition at NFS is provided. The 
design process for this experiment is outlined; including 
the design of the scenarios, determination of environments 
to be used; and the background of the subjects who 
participated m this project. A description of the voice 
training processes, practice sessions, and actual conduct of 
the experiment is provided. Analysis of the data from the 
experiment is provided in Chapter VI. Finally, conclusions 
from the experiment and the authors' recommendations for 
further study and applications follow the analysis of the 
ci a t a . 
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bPEECH TwCHNuLuGY 



II 



A INTRODUCTION 

Automatic speech recognition technology has progressed 
to the point where voice recognition products are being used 
in some applications* but further advances must be made 
before machines that recognize speech will become 
c uiTimonp 1 a c e. This chapter will discuss some terminology used 
in voice recognition* explain some general theory in ASR* 
review work that has been done at the Naval Postgraduate 
School in this field* and describe the equipment that was 
used in this experiment. 

B. DEFINITION OF TERMS 

A distinction has tu be made between recognition and 
under stand ing in automatic speech recognition. Most 
successful voice recognition machines today recognize short 
utterances of speech. An utterance refers to words or short 
phrases spoken with pauses between them. Recognition cf 
these short utterances using pattern matching techniques is 
pattern recognition* not und er s tand i ng . W. A. Lea defines 
voice recognition by machine "generally as the process of 
transforming the continuous human acoustic signal into 
discrete representations which may be comprehended to affect 
responsive behavior. “ CRef. 2: p. 40 D 
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T V, e technology required for recognition of general 
conversational speech exceeds current capab i 1 ities. The 
difficulty lies in the contextual properties of speech. In 
conversational speech* individual words often do not carry 
enough information for the word to be recognized by itself. 
Speech contains many acoustically ambiguous sounds* so that 
information needed to interpret a spoken word is not found 
in the word itself approximately thirty percent of the time. 
The information is spread out over several words rather than 
being contained in individual phonemes. A phoneme is a 
member- of the set of smallest units of speech. Thus 
contextual information processing may be required for any 
speech recognition beyond that of a limited number of 
carefully pronounced isolated utterances. Artificial 
intelligence and contextual information processing are key 
to the future development and advances of speech 
recognition. C R e f . i 1 

There are two major categories of speech recognition 
used. Discrete speech (isolated utterances) refers to words 
or snort phrases that are spoken with pauses between them. A 
gap of at least 0. 1 seconds of silence between utterances is 
required in order to distinguish between the boundaries of 
the utterances. Continuous speech refers to normal speech 
containing no word boundaries or pauses. Systems that take 
continuous speech and attempt to use semantic information 
are called understanding systems. Within both discrete and 
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continuous categories/ there are speaker-independent systems 
and speaker-dependent systems. With speaker-dependent 
systems* individual users must “train" the system with a 
given set of vocabulary in order for the system to recognize 




CURRENTLY FIELDED 

UNDER DEVELOPMENT 

Figure 2.1 ASR Systems 

their voices. Training involves users giving representative 
samples of their voice for each utterance. With speaker- 
independent systems# individual users are not required to 
train their voice patterns. They use the vocabulary already 
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-t or&d in memory which mag be used by any user At this 
time- there are no continuous s p ea i< er- i nd e p end en t systems on 

r k 0 market. 

C. PROBLEMS' OF SPEECH RECOGNITION 

The main goal of researchers in the automatic speech 
recognition field is to develop systems that will recognize 
speech with the greatest degree or accuracy; in the least 
amount of time; and at the lowest cost. The more advanced an 
automatic speech recognition system is; and the larger its 
vocabulary; the greater the challenge to develop more 
efficient methods to recognize voice inputs. One problem is 
how to combine storage and computation considerations in 
orcksr to optimize system performance. There are different 
storage or computational advantages in using sets of rules; 
procedures; templates; or networks to improve performance. 
Algorithm improvements aimed at efficient search strategies; 
ana arecomputing relationship networks are two methods of 
speeding up the recognition process; rather than using 
random or exhaustive search techniques. These methods can 
also improve recognition accuracy. 

Other problems are concerned with machine intelligence. 
These include how to represent and utilize contextual 
information; and how to make a c lass i f i ca t i on decision using 
advice from acoustic and semiotic (non-acoustic ) knowledge 
sources. CRef. 2 3 
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OVERVIEW OF SPEECH RECOGNITION TECHNOLOGIES 



I • Genera 1 

Automatic speech recognition equipment is composed 
of sound analyzers followed by word classifiers There are 
four steps in speech recognition which consist of a 
transducer-; signal processor; feature extractor; and 
utterance classifier. 

a. Sound Analysis 

The transducer and signal processor equate to 
the human ear and cochlea These are most commonly a 
microphone and spectrum analyzer. They take the voice analog 
signal and measure the relative intensities of the component 
sine waves. Acoustic information is represented by the time 
evolution of the power spectrum. The initial analyses 
produce parametric representat i ons of the speech. The 
function of the feature extractor is basically that of data 
compression. The parametric representation of the speech may 
contain redundant or irrelevant information which can be 
removed without losing any information. The removal of this 
irrelevant information reduces the computational load for 
all subsequent processing. [Ref. 33 

b. Classification 

The utterance classifier equates to the brain. 
In this step; internal models of acoustic patterns produced 
by speech are stored in memory. These reference patterns are 
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intsii; died ayainst the internal representations of the spoken 

0 1 t, erance 

2 Discrete Speech Recognition 

The differences between discrete and continuous 
speech systems become obscure when attempting to classify a 
particular approach as either discrete or continuous The 
most obvious characteristics of discrete speech are the 
required pauses between utterances and the speaking rates of 
users. Average speaking rates between thirty and seventy 
utterances per minute have been achieved by individuals 
using discrete speech in factory environments. [Ref. 4: p. 

1 74 3 

The most elementary and widely used approach to 
speech c 1 a s s i f i c a t i on is the use of a two-dimensional matrix 
of the utterance.- with time along one axis and frequency 
energy content along the other axis. Reference matrices 
whicn are generated for each word in the vocabulary during 
user training are compared to the matrix of the incoming 
speech. This comparison is typically performed by computing 
the sum of the square of the distances between corresponding 
matrix elements in the reference and unknown matrices. [Ref. 

1 J 

There is commonly a distortion between an utterance 
and its reference. This is partially due to a non-linear 
warping of the time scale/ caused by the word being 
pronounced at a different speed than its reference. To find 
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t n t optimal warping of the time scale/ the speech is divided 
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length* with the beginnings and ends being matched. [Ref. 5: 
p 26. 10. 1 3 

One of' the limitations with template matching is 
that Doth storage and computation grow linearly with the 
size of t h e vocabulary. Discrete speech systems are 
typically limited to 50 to 200 utterances/ though some may 
have a vocabulary as large as 300. Discrete voice systems 
generally have a high accuracy rate (greater than 95 
percent )# although the accuracy is partially dependent on 
the size of the vocabulary* simularity of the sounds of the 
vocabulary words* and whether or not the system is speaker 
dependent. The majority of fielded speech recognition 
systems are discrete voice machines because the technology 
is simpler to produce and operate. 

3. Continuous Speech Recognition 

The most obvious characteristics of continuous 
speech are the naturalness of the speech and the speaking 
rate. Without artificial breaks in speech* speakers can talk 
at a rate of about 150 to 300 words per minute. CRef. 2: p. 
£6j In continuous speech* it is difficult to determine 
where one word ends and another begins. Ad d i t i ona 1 1 y * the 
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5co patterns of words exhibit a greater variability 
d spend in g on the context in which they are used. Vocabulary 
size has a big impact on the complexity of these systems. 
Techniques for compact r e p r e s en ta t i or, of acoustic patterns 
of words# and techniques for reducing searches by 
constraining the set of possible words that can occur at a 
given point assume added importance. 

Dynamic programming is another method of time 
alignment that is used in discrete speech recognition# but 
also greatly enhances the capability of continuous speech 
recognition. Dynamic programming allows for nonlinear 
translation of speech# thus achieving better interior- 
matching of the words. It determines how to time align the 
segments of the reference and spoken utterances. The process 
computes many more combinations of time alignments between 
reference and spoken utterances than the static matrix 
method# but it significantly improves recognition results 
fo 7 multisyllabic utterances. Chef. 4: p. 1903 

Efficient search-path reducing algorithms can 
potentially reduce the computational burden of matrix 
comparisons. These algorithms are made possible by 
representing words as strings of phonemes. Two advantages of 
recognizing phonemes in a voice recognition system are (i) 
selective recall of word prototypes and (2) reduction of 
memory requirements to store word prototypes. Selective 
recall reduces the number of prototypes that need to be 
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processed By keying on phonemes and using them to form 
words* algorithms can be developed to process templates in 
an order more likelu to result in an earlier match. Thus the 
use of phonemes* or other subword prototypes* allows 
compression of the parametric representation of word 
prototypes and allows dynamic control to search only a 
subset of the word prototype dictionary. LRef. 3; p. 453 As 
L? r! White states [Ref. 3: p 493 “A solution to the 
fundamental problem of local acoustic ambiguity is to have 
models of speech sounds at all levels (phoneme* syllable* 
word* phrase)* and a strategy allowing lower models to call 
on higher models to resolve ambiguities and a strategy 
allowing the higher models to call on lower models to 
request further analysis. “ These methods* which are not 
available yet, would increase accuracy and conserve 
c omp utational resources. 

A method used in current continuous speech 
recognition equipment is the use of grammars to limit the 
number of utterances from which the machine must choose 
based on previously recognized utterances. A grammar is a 
rep re sen tat ion of allowable word sequences in a state 
diagram composed of nodes representing a word or group of 
words and the possible transitions between nodes. This 
method is very efficient in a system with a limited 
vocabulary* and in which strictly formatted sequences of 
words are required. LRef. 2: p. 523 



E. PREVIOUS EXPERIMENTS CONDUCTED 

There have been a number of studies and experiments 
conducted at the Naval Postgraduate School concerned with 
the applications of using voice recognition technology in 
command ana control tasks. Some of these experiments and 
outcomes of r elated studies will be reviewed briefly here. 

Tne objective of one experiment was to try to determine 
the possibility of achieving speaker independence using 
speaker dependent voice recognition equipment. The results 
showed about 99 percent accuracy when the subject's voice 
patterns were in memory along with those of four other 
users> and approximately a 95 percent accuracy when the 
subject's voice patterns were not in memory/ but those of 
four other users were. In this experiment# ail subjects were 
considered experienced in using the voice recognition 
equipment [Ref. 63 In a follow-on to this# another 
experiment tested the recognition accuracy with first time 
(naive) users of voice recognition equipment. Again the 
patterns of four- independent speakers were in the memory of 
speaker dependent equipment. In this case a 96.85 percent 
recognition accuracy was achieved# showing that training and 
practice may not be required for voice applications. [Ref. 
73 

Another study was made to determine the effects of 
feedback on the performance of voice recognition equipment. 
Feedback is used to indicate if a misrecognition or 



24 



ncnrt'cognition occurs. It was found that feedback has a 
limitec effect on performance. Subjects not accustomed to 
feedback reduced errors by five percent when feedback was 
introduced; while subjects used to feedback had about five 
percent more errors when feedback was reduced. [Ref. 8] 

One study attempted to determine the effect of operator 
mental loading on voice recognition performance. Besides 
being required to repeat words presented audibly via a 
headsets subjects were subjected to various degrees of 
men cal loading by being required to make decisions of 
varying degrees of difficulty and make a response at the 
same time. A General Dynamics Response Analysis Tester 
(RATER; Plod el 3) was used to simulate the operator mental 
loading. Results showed that voice recognition performance 
degraded as mental loading was imposed on the subjects. 

L R t: f 9 3 

There were two experiments conducted to see if the use 
of masks degraded voice recognition. Discrete speech 
equipment was used in both cases. In the first; there was an 
iricr&a se in recognition errors made when using a 
stenographers mask; but data showed this was primarily due 
to the inexperience of some subjects in speaking into a mask 
CRef. 103. In the second experiment with Army protective 
(gas) masks; there was more serious degradation. The authors 
felt more research was needed in the placement of the 
microphone in the mask; and variation of word boundary 
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to help alleviate breathing sound problems 



w h i c h 



p arame t e r s 

usually occur at the beginning and ending of uords. [Ref. 

i 1 3 



There have been two experiments comparing the use of 
voice recognition equipment and manual typing to enter 
commands into a computer. In both cases discrete speech 
recognition equipment was used. In the first experiment; 
sue- jeers followed a fixed scenario of instructions in which 
they accessed the ARPANET and then performed a prescribed 
set of tasks The ARPANET is a large distributed network of 
computers which are located around the United States and 
other countries. Prior to this experiment; subjects only had 
an average of three hours of training on using voice 
systems. Each subject performed the scenario four times with 
each method of input; with half using voice input first and 
the other half using keyboard entry first. A secondary task 
was assigned in which subjects were to use any free time 
curing the sessions to transcribe some data by hand onto a 
data sheet. The results of this experiment showed that: 

(1) Voice input was 17.57- faster than manual typing input. 

(2) Manual typing input had 183. 2% more entry errors. 

(3) Voice input allowed subjects to transcribe 25% more 
information than during manual input. 

The results were all statistically significant (p < .05) and 
demonstrate that it is feasible to use current commercially 
available voice recognition equipment to run standard 
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on an ARPANET type network; with minimal 
additional training. CRef. 123 

In a separate experiment/ subjects used voice and 
hsyooard methods to input commands in a computer-aided 
war game. The Warfare Environment Simulator (WES) was used to 
simulate a naval warfare environment. In this thesis work/ 
W. J. iicScriey exercised twelve subjects with a set of 
tupical WES commands using voice entry; and compared the 
speed ana error results with those of keyboard entry. Based 
on hi s Gata, NcSorley concluded that subjects were able to 
input commands faster and with fewer errors using the manual 
typing mode than with unbuffered voice entry. CRef. 131 

The final work leading up to this thesis was done by 
•Jo nr; Lombardo His effort was to show that continuous voice 
recognition equipment could also be used to input commands 
in a computer-aided wargame. Lombardo's thesis was directed 
towages the analysis of the strictly formatted command 
syntax of the Naval Warfare Interactive Simulation System 
* NWISS) ; a wargame that will be described briefly in the 
next chapter. He then developed continuous voice application 
software which allowed voice input of NWISS commands in 
unbroken phrases. It was this software that was utilized for 
the continuous voice input in this thesis. CRef. 143 
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F DESuR IPTluN Uh- HARDWARE UTILIZED 

1 Discrete Speech Recognition Equipment 

The Threshold Technology/ Inc. Model T6Q0 voice 
recognition system (hereafter referred to as the T 6 QO 1 or 
Thresnoid ) uas used in this experiment to input discrete 
voice commands. With several added memory devices/ the 
system had the capacity to store up to 256 utterances 0. 1 to 
2 seconds long. The system also contained a magnetic tape 
cartridge unit which allowed the subjects to record their 
individual voice patterns and NWI5S commands after they 
initially trained the machine. All subjects were assigned 
individual tapes. Then/ when the subject came back to use 
the T600 for NWISS sessions/ the cartridge was read back 
into memory and the subject was ready to give voice input 
commands. Gain could be manually set to help overcome the 
interference background noise. A Shuts SM10 headset 
microphone; supplied as standard equipment with the T600/ 
was used as the input device. 

In this experiment/ we used the unbuffered mode 
which means that as the voice recognizer accepts a voice 
input/ the word or phrase defined for the utterance it 
recognizes is displayed on the screen/ and an ASCII 
character stream is immediately sent to the host computer 
without any verification by the user that the voice input 
was correctly interpreted. This allows for the possibility 
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that the voice recognizer identifies something other than 
what uou actually said/ and therefore transmits the wrong 
ASCII stream. However/ the voice input stream is retained in 
a buffer of the host computer until a complete NWISS command 
is entered and a carriage return character is issued to 
instruct the computer to act upon the order. Until that time 
the user has the option to delete portions of the input 
stream and repeat the desired input/ or to cancel the entire 
stream by issueing a “Control K' 1 character and starting over 
again. [Ref 123 

2 . Continu o us Speech Recognition Equipment 

The Vertex 3000 voice terminal was the system used 
in this experiment to input continuous speech commands. The 
system is designed with a feature to automatically set the 
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over as many as twenty grammars. If the vocabulary is 
contained in a single grammar/ voice input can be in the 
form of a naturally spoken stream of words; numbers; or 
phrases/ with no pauses. An example of this would be the use 
of continuous voice recognition equipment to input zip 
codes. However, if more than one grammar is required/ a 
pause in speech input must be made when switching from one 
grammar to another. There is a finite limit to grammar size, 
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&fis»eil an the total number of words and the complexity of the 
node transition network. This is necessary to allow the 
system to remain ,! real time 1 ' in terms of computation speed 
and scored voice pattern requirements. Thus at any instant 
the system is dealing with a subset of the entire grammar. A 
Shore Sri 1 2 A headset microphone. supplied as standard 
equipment with the Vertex, was used as the input device. 

In addition to the Verbex 3000. the system includes 
the Speech Application Development Systems (SPADS) which 
allows the user to program the voice terminal to run a 
particular application. instead of purchasing customer 
engineering services from Verbex. It is this system that 
allowed Lombardo to develop a program to use in applying 
voice commands to NWISS. CRef. 14] 

One feature of the Verbex 3000 is a small unit that 
prompts the user for the proper command. Thus when the NWISS 
game is brought on line, it prompts the user to enter an 
NWISS command. When the first phrase is entered by voice. 



t it © 


system 


searches 


for the 


proper grammar 


. and then 


prompts 


c h e 


user 


aga i n to 


enter 


any of an a 


liowable subset of 


p h r a 


s e s to 


continue 


or complete the order. 


When the 


c omman d 



is completed, it is automa t i ca 1 1 y transmitted for execution, 
and the system again prompts for a new command. As with the 
T 600; the Verbex user may cancel a partial stream by saying 
"Control K" and starting over. 



30 



5. La&oratorq Equipment 

The remainder of the equipment used in playing the 
wargame mere keyboard terminals and the VAX 11/780 computer, 
on which the NWISS software resided. The Vertex and 
Threshold speech input systems communicated to the VAX 
through Sxegler ADM-31 terminals. A Digital VT102 terminal 

V 

was used as the manual keyboard input device. Digital VT100 
terminals were used at all positions as status boards for 
the NWISS sessions. Ramtek high resolution color graphics 
terminals were used to display the game scenarios to the 
subjects at each position. The configuration of the lab is 
e escribed in Chapter vA with a figure of the lab layout 
provided. 

A cassette deck was used to play a tape to introduce 
noise during the NWISS sessions. Eight inch, eight ohm 
speakers were placed at each position. Random crowd noise 
was recorded on tape at a gathering of about thirty people 
tailing conversationally in an auditorium. 

<?. SUNriARY 

In this chapter we have discussed some commonly used 
terms and have described differences between discrete and 
continuous modes of voice input. We have reviewed some 
general theory used in design and development of automatic 
speech recognition systems. Previous experiments with voice 
technology conducted at the Naval Postgraduate School were 
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scribed in order to give some background on uhat led up to 
is experiment; and possible uses and limitations of' voice 
eh no logy applications in the military. Finally/ a 
script ion uias provided of the actual speech machines and 
sociated equipment used in this experiment. 
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THE NAVAL WARFARE INTERACTIVE SIMULATION 5VSTEH 



A INTRODUCTION 

This chapter contains a brief description of NWISS and 
why it was used in support of this experiment. It also 
includes the scenarios developed for the experiment/ their 
installation/ and the measures of effectiveness selected 
The methods used to collect data and compare the three 
command input technologies are also described. 

H wHV hi id I b3 WAS USED 

NWISS is/ as its name implies/ an interactive simulation 
of naval warfare. Its mission was originally to train senior 
Naval officers in force-level tactical decision making and 
manag esnen t of command and control CRef. 15: p . i — ID. At NFS 

NWISS is resident on a VAX 11/780 computer. The peripheral 
VT100/102 terminals and RAMTEK graphics terminals provide 
i np ut / output modules when grouped together. Voice equipment 
was used as "front-end" devices in two of the modules. All 
inputs and outputs can be accomplished bu one trained 
subject from one module. NWISS was loaded/ controlled and 
monitored from a fourth module during these experiments. 
The purpose of NWISS is to stimulate a stressful environment 
that allows the user to intervene through use of NWISS 
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a large 






commands to change the view of the tactical 



icatular u o f 

situation in the battle area< cause changes in the course 
and speed of vehicles in one force; and launch air assets 
with user defined weapon loads and missions. NWISS also 
displays the status of the user's forces on a variety of 
menu selected displays. For this experiment; each station 
consisted of a command input terminal; a force status 
terminal and a video monitor to graphically display force 
disposition and identity. 

NWI3S was used to support this experiment because it 
offers the ability to present identical tactical situations 
to three isolated subjects at the same time. This allows a 
side by side comparison of the command input methods chosen 
by the isolated subjects when confronted by identical 
tactical stimuli in a controlled environment. It was also 
selected because it is available at NPS and a pool of 
trained operators was at hand. 

’The most important criterion; though; is NWISS's large 
vocabulary. If a very small vocabulary is chosen to run an 
experiment; the subjects may become bored through repetition 
of the same commands. Approximately 100 words that are 
necessary to conduct a variety of missions with all or part 
of a carrier battle group were selected for the voice input 
technologies. Twenty commands related to control of the 
tactical force display were also included. Tables of the 
complete vocabularies for the three input technologies are 
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included in Appendix A. These vocabularies were determined 
in advance and were limited by the size of the memory of the 
Vertex system. Using the forces and commands in these 
subsets of the NWISS dictionary/ four scenarios were 
designed to stimulate the subjects. 

In all four scenarios/ the Orange and Neutral force 
actions were preprogrammed as were the Orange defensive 
poseurs and radar emmisions control (EMCON) status. No 
Orange aircraft were used because introducing them is more 
complex than required for this experiment. The three views 
of one world for the Blue forces were on approximately the 
same latitude and separated in longitude by approximately 
500 nautical miles. 

Since NWISS is limited to a maximum of six sets of 
^orces/ the Orange and Neutral forces were structured as one 
\ i e L; each. There were three separate views for the Blue 
forces which allowed the subjects to control their own ships 
and planes. The details of the process of assembling these 
various views and overlapping them are not included in this 
paper. They are explained in some detail in an NFS thesis 
by Owens and Brown. CRef. 16: pp. 25-34D 

C. SCENARIOS 

A total of four scenarios were designed so that the 
subjects would be placed in situations that required use of 
the available NWISS vocabulary and to stimulate different 
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responses This section describes the basic opening force 
positions for each of the scenarios and the military 
situation briefing for that scenario. The situation 
briefings given to the subjects are in Appendix E. 

1 . Scenario A 

Scenario A includes twelve P3C aircraft/ an aircraft 
carrier/ two frigates/ and an attack submarine for the Blue 
forces. Grange forces consist of a carrier and six surface 
combatants. The Orange forces are escorting eight Neutral 
merchant ships in a convoy that is moving south as shown in 
Figure 3 1. In this scenario/ hostilities have Deen 
declared between Blue and Orange; and the objective is to 
identify and destroy the Orange escort vessels without 
striking the merchant ships. The merchant ships are 
reportedly carrying nuclear and chemical weapons to an 
Grange ally. 

2. Scenario B 

Blue forces for Scenario B are an aircraft carrier; 
two P3C's to provide targeting inf orma t i on; and two flights 
of five attack aircraft. Twenty merchant vessels are at 
anchor outside a fictitious harbor facility waiting to 
unload their cargo. Four Orange surface combatants are 
positioned in a line between the Blue forces and the 
merchant ships to defend them from attack. In this 
situation; hostilities have not been declared between Blue 
and Grange although tensions are high. Blue's mission is to 
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LEGEND: Blue Aircraft 

(3 Blue Surface Combatant 
^ Blue Submarine 
| | Neutral Merchant Ship 

0 Orange Surface Combatant 




1 5 4 E 1 5 5 E 156E 

Figure 3.1 Scenario A 
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ust :h$ airbor-ne aircraft and the remaining carrier based 
planes to conduct a strike on the merchant vessels The 
rules of engagement do not allow the Blue forces to fire on 
Orange until Orange has fired on them. From the opening 
positions as shown in Figure 3. 2i neither the merchant ships 
nor the Orange combatants are visible to Blue/ but the 
mission is directed to fly to the north. 

3. Scenario C 

Scenario C presents a benign environment in which 
Blue tasking is to locate and identify as many objects as 
possible and designate them as Friendly; Neutral or Enemy. 
Blue has one P3C and four F14A aircraft to accomplish this 
mission. A total of thirty ships and submarines are 
scattered over an area approximately 400 x 500 nautical 
miles as shown in Figure 3.3. Combatants for both Blue and 
Orange as well as several merchant ships and fishing boats 
are in this rectangle. The subjects were told that they did 
not have control of the Blue surface forces in their area. 
This was an artificially introduced constraint to complicate 
the tasks of location and identification. 

4. Scenario D 

The final situation pitted two carrier battle groups 
of approximately equal strength against one another. 
Blues's task was to locate and identify the elements of the 
Orange battle group. Hositilities had not been declared but 
the subjects were given wide discretion to respond to any 
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Hostile acts by Orange The opening positions in Figure 3.4 
place the Orange carrier battle croup to the north of the 
Blue group with Orange headed north and Blue to the 

n o r t h w e s t . 

D. MEASURES OF EFFECTIVENESS 

The subjects were familiar with the play of NUJISS and 
.Had experienced shortcuts in operating the game. Some 
commands such as changing the scale of the display are easy 
to accomplish and rapidly achieved. This is one of the 
reasons we chose to tell the subjects a false set of 
measures of effectiveness. When the subjects received their 
mission briefing prior to each session* they were told some 
of the measures of effectiveness by which their success or 
failure would be determined. Included were such things as 
the number of merchant vessels sunk or damaged in Scenario 
B * tne numiber of ships correctly located and identified in 
Scenario C> and the number of Blue forces lost to hostile 

fire in Scenario A CSee Appendix BD. In each case* the 

« 

sub jeer was urged to accomplish the goal before time 
expired. These MOE ' s were not used for analysis* but were 
intended to stimulate the subjects to try to achieve a 
mean ingful goal. 

The actual measures of effectiveness are related to the 
number of NWISS commands entered and their accuracy. The 
number of commands entered is the most positive indicator of 
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mastery of the input technique. We felt that the four 
^eerier ioc would provide the opportunity to use many of the 
commands available in an environment complicated by the 
different light and noise levels. Time pressure was 
impressed on the subjects by the schedule and by the actions 
of the facilitators. Since all the trials lasted the same 
nurn&sr of game minutes; tne number of commands entered is 
essentially a rate of entry. The number of commands entered 
could nave been increased significantly by the savvy subject 
by changing the display radius numerous times. The briefing 
which stressed fictitious MOE's effectively removed this 
type of performance. Further; if ail that was desired for 
data analysis was the rate of data entry; very little 
experimentation is needed for the keyboard station. That 
Tats can be calculated from a representative typing speed 
ar.d error rate. We were interested in the environmental 
effects on the input methods in a command post setting; 
where actual commands to forces are not necessarily given in 
a continuous stream. 

Accuracy can be measured to some extent by the number of 
times the "Control K" command is used to delete an attempted 
command. However; there are some limitations to using 
"Control K" as a measure of the number of errors made by 
subjects. Since it aborts the current command and does not 
execute it; the "Control K" could be viewed as a change of 
mind on the subject's part. From experience using the NWISS 



g cd a) e , tnoughi it is just as easy to enter a com pie tea 
command and issue a subsequent correction as it is to abort 

1 1 . 

There are also three equipment related factors that 
ce tract from the use of “Control K “ as an error counter, 
first, in the continuous speech recognition system, after 
t!:e keyboard has Deen used to enter an NWISS command not 
included in the ill word vocabulary* the “Control K M command 
:* s necessary to return to the beginning of the command input 
sequence Therefore, this use of “Control K" does not 
indicate an error, but the correct use of the system. 

Second, it could be that the subject's utterance was not 
recon n i zed by the system and the subject elected to enter 
the command manually. The result is the same as in the 
£ irs t case: the abort command must be used to return to the 
stare of a new command input sequence. 

Finally* both the Threshold aiscrete recognition system 
and the keyboard have the capability to backspace and 
correct errors in any command without deleting the whole 
command. So it is possible that all the subjects in these 
two groups would not need the abort command except to 
indicate a change of mind. However, using the backspace 
function to make corrections involved using time that could 
otherwise be used to perform a “Control K“ and reenter the 
command. With all these factors considered, the “Control K“ 
command provides the best measure available of errors made 
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in entering commands; although it may not be as strong a 
measure as the number of commands entered. 

e data collection 

Data collection was accomplished through several utility 
programs of the VAX 11/780. All commands entered by the 
three subjects'' command terminals were sent to separate 
files in memory as was the record of the umpire's game 
control terminal. This process set up a record of all 
activity by the subjects and kept a h i s t o r i c a 1 account of 
all engagements* weapon firings and resultant damage 
displayed on the umpire's terminal for further study. 

The data files for each of the input methods were then 
filtered by another VAX utility program to output the 
“Control K“ abort commands. The printed outputs were then 
Hand counted to determine the number of commands entered and 
tne number of errors made. 

One additional comment is necessary regarding how the 
number of commands entered was determined. The aircraft 
launch sequence is widely used; and is the most lengthy and 
difficult command. Since the aircraft launch sequence 
involves at least five lines to be complete and could be 
aborted anywhere within those lines; each line of the five 
that was correctly entered was counted as a command entered. 
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F SUMMARY 

This chapter has discussed un y NUilSS was used for this 
experiment including a brief description of NWISS 
capabilities. The four scenarios presented to the subjects 
are described/ figures of the opening positions for all 
objects are given/ and the situation briefings for the 
subjects are discussed. Finally/ the measures of 
effectiveness and data collection techniques are outlined 
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IV. 



VOICE EXPERIMENT BACKGROUND 



INTRODUCTION 



In preparing for the N W I Ss voice experiment/ many 
ops had to be taken into account. The background of the 



sub 


j acts used in 


the experiment 


(their service/ 


g an) ing 


e x p 


enence, and 


experience 


with 


voice equipment 


and 


with 


NW i 


SS ) had to be 


considered. 


Other 


decisions made 


i n 


the 



experimental design stage were concerned with time periods 
for tne experiment/ the numoer and length of runs/ and 
randomness of sequences/ of environments/ and of scenarios. 

it background of subjects 

The background of the subjects participating in the 
voice experiment was extensive and varied. All subjects were 
students in the Command/ Control, and Communications CC3) or 
the Space Operations curricula. There was a mix of A*rmy<9>, 
Navy (21)/ Air Force (7)/ Marine(l)/ and civilian(l) students. 
Thirty four were male and five were female. 

All students had one year of graduate level course 
instruction. Common courses included 0S3404 (Man-Machine 
Interaction)/ 0S3603 (Simulation and Wargaming)/ and 0S4602 
(C3 Systems Evaluation). In 0S3404 students were introduced 
to continuous and discrete automatic speech. recognition 
theory/ and experimented with discrete voice inputs. They 
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were also introduced to the Digital VT100/1G2 keyboard 
teriiiiiicls and Ramtek graphics screens in the Ular Lab which 
would be used to play the war games. 

DS3603 included instructions on gaming theory and 
experimentation. Students were introduced to NWISS; 
including its background and symbology. Further instruction 
on discrete and continuous voice was given; and all students 
trained on either Vertex (continuous) or Threshold 
(discrete) machines in the War Lab using a prescribed set of 
NWISS vocabulary. They had two practice sessions (three 
hours total) of entering common NWISS commands using the 
equipment they had trained. It should be noted that the 
entire NWISS vocabulary could not be trained because of 
equipment and design limitations. The course also included 
three 3 hour sessions of groups playing NWISS with a Sea of 
Japan scenario, using keyboard to input commands. 

In QS4602; students had additional instruction on 
experimental design and analysis. They also had seven more 
three hour NWISS sessions in conjunction with another 
experiment. Prior to the actual start of the voice 
experiment; subjects were assigned to one of the methods of 
input (Vertex; Threshold; or keyboard); with a rough balance 
of naval and non-naval students at the three positions. Each 
had an additional one hour practice session using his or her 
assigned method of input using a practice scenario in order 
to regain familiarity with the respective methods of input. 
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Across all courses of instruction; then; the total amount of 
t raining related to this experiment was greater than sixty 
hours per student. 

C ENVIRONMENTAL CONDITIONS 

The experiment was designed to be run in a variety of 
environments; based upon four different combinations of 
light and noise conditions. Ad d i t i ona 1 1 lj ; four different 
scenarios were designed to be run so that subjects would 
have a different scenario during each of their four run- 
f or ^record sessions. This design was chosen to complement 
the training and to further reduce the impact of learning on 
the resultant data. Otherwise; the data would be affected 
as subjects gained familiarity with a scenario. learning 
from past mistakes; and thus increasing the numerical value 
c f tne measure of effectiveness. 

The four combinations of environment are as follows. 

1 - low noise; normal lighting; (benign condition) 

2 - low noi.se; low lighting 

3 - high noise; normal lighting 

4 - high noise; low lighting; (most stressful condition) 
These conditions were set to be representat i ve of the 
environment in a military command and control center; or in 
other military settings where decisions have to be made and 
orders issued. The experiment was designed to compare 
continuous voice; discrete voice; and keyboard methods of 
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input; ana to test if there was an impact on input frequency 
for any of the three methods as a result of different 
environmental conditions. 

Normal light conditions were defined as having overhead 
^lucres cent lights off; with track lights turned on and 
directed downward at a forty five degree angle toward the 
nearest wall. Light measured near the game terminal screen 
was approximately 1.0 f oat-lamb er t. Low light conditions 
were set by turning out all overhead lights; resulting in 
approx imately 05 f oo t-lamo er ts of light provided by the 
game terminal CRT's and the Ramtek graphics screen. 

Noise was introduced by playing a tape of background 
conversations of a large group of people. Speakers were 
placed beside the game terminals at all three positions. Low 
noise was set at a 65 decibel level; and high noise was set 
at an 85 decibel level; as measured on a C-scaie audio meter 
held at the subject's ear level. See Figure 5. 1 in Chapter V 
for the lab conf iguration. 

D SCHEDULING OF SESSIONS 

The voice experiment was designed to be run during lab 
time reserved for the 0S4602 class. Students in the class 
were assigned to one of three sections based on their course 
schedule. This process was done by school academic 
schedulers; with no input from the personnel involved in 
this experiment. There were 39 students in the experiment; 
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The authors of 



earn assigned tc one of the three sections, 
this thesis were net subjects due to their familiarity with 
the scenarios and hOE; which could affect the validity of 
the outcome of the experiment. 

The actual split of students resulted in two sections of 
twelve people each and one section with fifteen people. Each 
section was scheduled for two 3 hour blocks of lab time per 
wesK. The subjects were randomly assigned by an associate of 
the laboratory to groups of three (one Threshold/ one 

verDex; and one keyboard operator) within each section. This 
resulted in thirteen groups of subjects. Two groups were 
s c r » e a u I e d for each of five lab periods/ with three groups in 
tr. e sixth lab period each week. Each group was scheduled for 
one hour sessions during which each subject would conduct 
pis or her own individual NWISS run. This allowed for 
actual running time of forty minutes each/ leaving twenty 
minutes for closing each session/ setting up the scenario 



an u stations 


for the next 


run/ 


and briefing 


the scenario 


to 


tne next group 


of sub jects. 


The 


schedule for 


the groups 


of 


subjects and 


respective 


scenario and 


environments 


i s 



provided at Appendix C. 

This schedule allowed for four r uns-f or-r ecor d for each 
group; resulting in fifty two total sessions/ each session 
providing a record from each of the three methods of input. 
There were thirteen runs for each different scenario/ and 
thirteen runs for each combination of environment. The 
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number of runs was limited bu the class schedule* the amount 
ut availaole lab time* and the number of subjects. 
Ultimately; 156 individual records were accumulated. 

E SCENAR ID/ENVIRONMENT SEQUENCE 

Randomness of student assignment and scenario/ 
environment sequences was given a high priority to ensure 
that data outcome was not biased. The assignment of 
personnel to groups was completely random; the only 
constraints being that students had to be assigned to one of 
the groups within their section; and people assigned to a 
Threshold or Vertex position had to have trained and 
practiced with that particular voice apparatus. Everyone* 
including the keyboard subjects* trained on either Vertex or 
Threshold. 

As stated earlier* each group of subjects underwent four 
test sessions* with a different wargaming scenario and 
noise/ light condition in each session. The order of the 
scenarios was completely counterbalanced for sequential 
position and both preceding and following treatment effects 
E Ref. 173. The order of the conditions was also 
counterbalanced using the same method. The combinations of 
scenarios with conditions were also c ount erba lane ed so that 
each scenario was paired with each condition; and so that 
each of these pairs appeared in each position (first; 
second* third; fourth); again with counterbalanced preceding 
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and following treatment effects 

Since nine of the original 48 subjects did not take part 
in the test phase of the e x p er imen t , the counterbalanc ir,g 
scheme was slightly compromised in that one scenario and one 
condition occurred one extra time in each sequence position 
(four times versus three times each). We believe this 
compromise was inconsequential/ especially since it was 
identical across all input methods/ and therefore would have 
r, o bearing on the outcome of the data. 

F. CONSTRAINTS 

There were a number of constraints which dictated part 
of the experimental design; some of which limited the scope 
of me experiment. The experiment took place in the C3 Lab 
at the Naval Postgraduate School. Because the lab is secure; 
a no because the NWISS program was already running on the 
lab's computer system; the C3 Lab is an ideal location in 
which to run military war games. However; there is dg way to 
soundproof different sections of the lab. As a result; the 
Keyboard operator could overhear the voice input subjects in 
the low noise environment. 

While there are other wargames available; NWISS is the 
only one at the Naval Postgraduate School that has the 
software available to allow use of the Threshold and Vertex 
equipment for input of voice commands. The Vertex and 
Threshold equipment are the only speech systems available in 
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the lab. Thus the experiment was constrained to using those 
■ iuc systems with the NWISS scenario. 

The scope of the experiment was also limited by time 
constraints. It had to be conducted at a time when the lab 
could he reserved; with no other experiments or projects 
being conducted. This limited the duration of the experiment 
to one month. Additionally; sessions had to be scheduled 
during regular hours for the GS4602 course so that there 
wou.d be no conflicts for students with other class 
schedules. Thus we were limited to four actual sessions per 
.. £ j e e t > with each session lasting approximately one hour to 
a i m w time to brief subjects and collect sufficient data. 

As a result of the time constraints and availability of 
bUDjsctsi we felt it was best to have subjects input 
commands using only one of the specified input mode stations 
so mat they would be as proficient as possible on their 
assigned device. There was not sufficient time to train and 
test everyone on all input devices. 

q 5 1 j hj pi a p y 

This chapter discussed the background of the factors 
usee in designing the voice NWISS experiment. The background 
of the subjects used in the experiment was outlined; 
discussing their experience in experimental procedures/ and 
familiarity with the Verbex and Threshold speech recognition 
systems; as well as with the keyboard and graphics terminals 
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in :he iaij. The design and definition of the noise and light 
environments for the experiment were described. The 
set ecu ling of test sessions was determined by the number of 
students available for the experiments as well as by their 
class schedules. Randomness plays an important role in this 
type of experiment in order to preclude biasing of data. 
Thus trie order of scenario and environment sequences for the 
different groups; as well as the assignment of students to 
i r, out devices; was all done in the random manner described. 



The chapter concluded with 
constraints which a f f e c t e d the 



a description of some of the 
design of the experiment. 
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V. 



CONDUCT OF T HE EXPERIMENT 



- introduction 

The previous chapters have dealt with voice technology 
a t 5 a the experimental design for this thesis. This chapter 



Will C 0 s 


C r 


i b e 


the actual 


tr a i n i 


ng and practice s 


essions that 


T 0 Ok pi 


a c 


e i 


the lay out 


and 


control of the 


laboratory/ 


c n c u c t 


of 


the 


e x p er imen t 


; and 


the method used 


in saving 


rsc; . ras 


0 f 


t h e 


individual 


r u n s . 







L LABORATORY CONFIGURATION AND CONTROL 
1 C onf iquration 

The experiment took place in the secure Command and 
Control (l-JAR) Lab at the Naval Postgraduate School. The lab 
; rouse i callu paneled and was partitioned into three bay 3/ 
each approximately sixteen by eighteen feet. The equipment 
for each of the three input modes was located in one of the 
sections by itself. The keyboard entry station was in the 



cental bay (see 


fig. 


5. 1 ). 


Each 


station 


had a 


R a m t e k 


graph i c s screen 


1 oca ted in 


one 


corner of 


the bay/ 


with a 


speaker producing 


noise 


beside 


i t . 


On one side of the 


Ramt e k 



screen was. a VT100 game status terminal which was used to 
retrieve game information only. On the other side was a 
player terminal (VT102 for keyboard or ADM-31 for voice) 
which allowed commands to be entered into the computer. At 
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O SPEAKER 
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RAMTEK 



□ 

S 

□ 



PLAYER TERMINAL (ADM-31 OR VT102) 

STATUS TERMINAL (VTIOO) 

GAME TERMINAL 

VERBEX 3000 




AMPLIFIER AND TAPE DECK 



Figure 5.1 Lab Layout 
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r. h r. ,-oice entry locations the terminals generally functioned 
as pass through devices for character strings earning from 
the voice recognition equipment enroute to the computer. In 
certain instances/ however; their keyboards could be used to 
generate characters if there were problems with voice 
recognition. This layout enabled subjects to see and have 
easy access to all terminals by simply turning in their 
chairs. Also located in the same bay as the Threshold 
station; tout separated by a partition; was the VAX/NWISS 
control station. It was here that each NWISS session was 
brought on line; and the experimenters could monitor the 
progress cf the game. There was additional equipment located 
in each bay that was not associated with the experiment. 

2 . Co ntrol of the Lab 

Laboratory access was not restricted during the 
individual voice practice sessions. However; during the 
final group NWISS sessions and during the v actual test 
sessions; access to the lab was more restricted. A notice 
was posted on the front entrance to the lab indicating that 
the experiment was in progress; and requesting personnel to 
use rhe rear entrance to the lab. This helped prevent 
additional noise in the lab caused by the opening and 
closing of the pneumatic sliding door in the front of the 
lab o y the Vertex station; and also prevented unwanted light 
from entering the laboratory environment. Non-experiment 
users had to use the lab in whatever the noise an d light 
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environment was during the experiment so that the 
environment settings were constant throughout the sessions. 
Environmental conditions which had been adjusted and 
measured prior to each run were maintained by these 
p u t ions. 

C i E A I M I NG 

1 G e n e r a 1 

As stated previously* all students were initially 
-andomlu assigned to either the continuous or discrete voice 
recognition system located in the C3 (Ular) Lab. They were 
giver a briefing on how to use their respective ASR devices* 
and how to train the equipment to recognize their utterances 
of me prescribed NWI3S vocabulary (This was a review for 
the Threshold users. >. The briefing included a step by step 
demonstration of an actual training session* and written 
instructions on the process were provided to all students. 
Subjects then reserved time on the system in order to 
perform their individual training sessions. 

Z'. Environment During Training 

The light level during individual speech training 
sessions was the normal lighting conditions in the lab. This 
level of one to two f oot-lamber ts was approximately that 
which would be present during two of the four environmental 
conditions for the experiment. 
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The background noise tape was played during training 
sessions at a level of sixty five decibels. This was also 
equal to that of two of the four experimental noise 
conditions. Noise was introduced during training in order to 
scfiieve greater recognition accuracy during the experiment! 
where at minimum this level of noise w o u 1 d be present L R e f . 
1B3. 

2 Discrete Voice Training 

The process for training the discrete Threshold 
equipment was fairly simple. After the initial set-up of the 
equipment* the procedure consisted of entering ten passes of 
siCh utterance into the T600. The recognizer automatically 
averaged the passes into single templates for each 
utterance. This process was repeated for each of the 113 
assigned NWISS vocabulary utterances. This vocabulary 
listing is shown in Appendix A. Although there were standard 
p: ciTjptb provided on the vocabulary sheets* subjects could 
bud st i tube any utterance they wished for the prompt in order 
to make it easier to remember* thereby p er sona 1 i z ing the 
vocabulary. This could also be used to alleviate any speech 
recognition problems peculiar to an individual's speech 
patterns. Similarities in the pr onunc ia t i on of different 
commands was the most common cause of mi sr ec ogn i t i on of 
utterances. 

After training the vocabulary* subjects could test 
their voice recognition accuracy by repeating vocabulary 



60 



•jj o r d s If recognition took place, the word the system 
recognized appeared on the CRT. If a match was not found; 



t h a r- 
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wa s 


an a u d 


i b I e 


beep from 
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device. 
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the other- 


was 


kept 


as 
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something 


hap p ened 


t o 


the first tape. 



This training took about one hour per subject. 

4 Continuous Voice Training 

All continuous voice subjects were assigned unique 
:u: digit user identifiers for use on the Vertex 3000. This 
n ui 7 »D er must be entered during the startup on the system. The 
number was used during training to initially record their 
voice patterns on a disk pack in the system. During 
subsequent practice sessions and testing* subjects had only 
tc enter their identifier and their voice patterns would be 
recalled for use. 

Training on the Vertex 3000 consisted of two phases; 
isolated and continuous. In the isolated training phase, the 
subject spoke each of the ill utterances in the Vocabulary 
at least twice by itself. This vocabulary listing is 



p r o v i d e d 


in Append i x A. 


The words 


to 


be trained 


were 


p V 0 iTi p ted 


by the Verbex 


3000. As 


with 


d iscrete 


voice 


training* 


subjects could 


substitute 


any 


ut teranc e 


they 



wished for the prompt* as long as they used it throughout 
the training and experiment sessions. 
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In the continuous training phase/ up to three of the 
.riLi ,'iduai utterances were grouped together and spoken 
continuously. Again/ the phrases were prompted by the 
system. Each utterance was included in about twenty such 
groups/ and was therefore repeated about twenty times in 
different sets of phrases. The continuous voice training 
consisted of training phrases in each of the ten grammars 
described earlier. Some of the utterances were unique to a 
given grammar while others; such as “Control K“; were common 
to all grammars and had to be trained in each grammar. The 
continuous training took four to five hours per subject. 
LEef 193 

5. Retraining 

After the initial training/ some subjects discovered 
that they had recognition problems with certain utterances. 
Born speech recognition systems allow for retraining of 
individual words/ and the subjects could do so at any time 
throughout the period of the exercise. In some cases with 
misrecognition problems; subjects changed the utterance 
which they used/ while in other cases they simply retrained 
with the utterance they had previously used. 

Retraining on the discrete system is very simple. 
After reading their individual tape into the T60Q; subjects 
just indicated the word that they wanted to retrain. As 
before; the utterance was . repeated ten times. Then the 
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subject could retrain other words if desired. After 
retraining, the voice patterns were re-recorded on the tape. 

Retraining on the continuous system was more time 
consuming. After initializing the system and entering their 
identity code; the subjects were given a series of prompts 
to choose the desired mode of operation. They responded 
af f irma t i vs 1 y to the prompt to retrain vocabulary. The 
system then began displaying the individual utterances in 



entire 


vocabulary 


. The 


sub jec ts h 


ad to 


step through 


e one 


D y one 


until 


they came 


to the 


utterance they 


e a to 


retrain. 


They 


then retra 


ined 


the ind ividual 



utterance. At that time the system began displaying phrases 
to be trained containing that utterance with other 
voeaoulary words. If the utterance applied to multiple 
grammars; more phrases would have to be repeated. At the 
end of retraining the designated utterance/ the system 
a u t oma t i c a 1 1 y left the retraining mode. Thus/ if subjects 
wanted to retrain more than one utterance; they were 
required to go through the above process again for each 
utterance. The process took approximately fifteen to twenty 
minuses for each utterance to be retrained. Because of the 
amount of time it takes to retrain vocabulary; many of the 
vertex users elected not to retrain problem utterances; 
accepting the handicap incurred. 
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D PRACTICE SESSIONS 

Initi a l Individual Tasks 

After training their speech patterns* subjects were 
given written instructions on bringing up the NWISS game and 
initializing their respective ASR/ equipment. They were given 
a list: of approximately 45 separate NWISS tasks to perform 

Du issuing commands using voice input. While the lists were 
different for the continuous and discrete , inputs because 
force names and call signs were different* the general set 
of tasks was the same. The tasks were stated in such a way 
as to require the subjects to put them into NWISS format 
first/ and then enter the commands as a recognizable speech 
input. 

The subjects were to perform the NWISS tasks in two 
sessions totaling approximately three hours. They were 
instructed to try to enter a command by voice up to three 
times if necessary* and then to revert to keyboard entry if 
the speech recognition was still unsuccessful. In the first 
session* subjects were instructed to perform as many of the 
tasks as possible* noting those that they could not do 

entirely oy voice. They were asked to complete the list 

during session two* and then go back and try to perform the 
problem tasks from session one again. 

There were several purposes for the individual 

practice described above. It helped to identify problems 
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with recognition; so that subjects could retrain vocabulary 
as required. Additionally; it was useful in teaching 

subjects to take a general task and convert it to NUJISS 
format and then say it in acceptable voice protocol. This 
was especially important for Vertex users due to the 
requirement for pauses between phrases that relate to 
different grammars. Most users initially thought they were 
naving recognition problems; when in fact they were not 
pausing at the correct places. The experimenters 
o anticipated in this practice session; one on Vertex and one 
on Threshold; so that we would be better able to offer 
assistance to subjects having problems on either of the 
devices. 

The design of the experiment scenarios was enhanced 
b y the outcome of the individual training in that it allowed 
the identification of NWISS vocabulary that was not trained 
Threshold and Verbex users. There was an additional 
pro Diem with Verbex in that while an individual word may 
Have teen trained; it may not have been included in all 

grammars or phrases; so that it could be used in some 
phrases or commands and not in others. These limitations 
were taken into account at the scenario design stage in 
order to insure that all or most commands could be entered 
entirely by voice input. However.- because the tasks for the 
re sz scenarios were general in nature; subjects could not be 



65 



precluded from using a strategy that required them to resort 
to Keyboard entry at times. 

2 . NUIISS Practice Sessions 

a. General 

The subjects went through a practice NWISS 
session Defore the actual experimental testing. As described 
in Chapter IV; all subjects were assigned to either 
continuous voice; discrete voice; or keyboard entry 

positions. Due to conflicting scheduling in the lab/ this 
practice session took place approximately one month after 
the individual practice sessions. Some loss of learning had 
obviously occurred during this period. However; rapid 

recovery occurred in most cases. 

b. Purpose 

There were several purposes for this practice 
session The first was for a r e f am i 1 i ar i 2 a t i on and further 
practice with the voice recognition systems for the Verbex 
arid Threshold personnel. Since ail subjects had just 
completed a separate experiment using NWISS; it was also to 
refamiliarize the voice users with the names of the forces 
for which they had trained their voice patterns. In 
addition; this was the first time that keyboard subjects 
used the names of their forces; since they had previously 
only practiced at one of the voice terminals before being 
assigned to keyboard. All subjects had easy access to status 
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sc remembering 



beards which shewed their available forces, 
the names was not a problem after reasonable practice. 

This was the first time that the subjects had 
participated in an NWIS3 scenario individually. All their 
previous experience had been as a player in a group, where 
there was a division of duties to be performed at a station. 
An important aspect of the practice session was to introduce 
the subjects to the format and procedures that would be 
followed in the test scenarios. The sample scenario was 
.indicative of the types of tasks that they would be required 
to perform during the a c t u a 1 experiment. 

The practice session was also important for the 
experimenters in preparation for the actual experiment. The 
same format- that would be used in the test sessions was 
fallowed. Running the thirteen groups through a dry run 
ensured that procedures which would be used later were 
smooth, and verified that there was time to complete a 
session and gather resultant data in the one hour periods 
that were scheduled. This also allowed testing of the four 
environmental conditions that would be used, noting the 
correct positioning of stereo speakers and settings on 
rheostats to insure equal noise and light at all three 
positions. Finally, the practice session allowed advice to 
oe offered to subjects on problems they were still having. 
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c Preparation for the Practice Session 

Prior to the arrival of the subjects for each 
session/ the lab uas set up for the start of the NWISB run. 
All equipment was turned on and the NUIISS scenario was 
initialized and put on pause so that the game would not 
progress until subjects were ready. The voice patterns for 
the Vertex and Threshold users were loaded into the systems. 
At each of the three positions' game terminals/ a command 
was entered to "photograph" the screen. This allowed the 
system to save everything that was input by the subjects/ 
and included system responses to commands; prompts/ and game 
time postings. The master game terminal was also 
photographed to provide a log of the progress of the NWISS 
game The subjects were then brought in to begin the 
session. 

d . Conduct of the Practice Session 

Prior to the start of the practice session; the 
subjects were given an a dm i n i s tr a t i ve briefing pertaining to 
the practice run and concerning instructions for the test 
sessions. The exact items covered are shown in Appendix D. 

Following the administrative briefing/ the 
subjects were briefed on the practice scenario. Each was 
given a three page handout. The first page stated the 
situation/ mission/ and apparent MOE's for the scenario (see 
Appendix E). The second page listed a number of sample tasks 
that the subjects might want to use in performing the stated 
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miss:. on (see Appendix E). The last page was a classified 
sneer providing the capabilities of sonrie of the forces/ 
weapons/ and detection systems used in NWISS. Because of its 
classification/ this page is not included as an appendix. A 
verbal briefing on their mission was also given to subjects 
and any questions were answered at this time. Subjects were 
reminded to turn the handout in after finishing the session/ 
and were instructed to go to their positions to start the 
exercise. 

e. Completion of the Practice Session 

The NWISS game was allowed to run for 
approximately forty five minutes. At the completion/ the 
subjects departed and/ if another group was to come in/ the 
experimenters prepared the lab for the next group. 
Additionally; copies of the photographed game screens were 
PT irteo out. In order to conserve paper; this was only done 
for the first few sessions so that we could verify that we 
were getting the correct information; and to determine how 
long the entire process would take. Printouts were not 
needed of ail the practice sessions because the data from 
these would not be used in the experiment. 

E TEST SESSIONS 

The same procedure was followed in the test sessions as 
was used in the practice sessions. Prior to the arrival of 
the subjects/ the proper NWISS scenario was initialized and 
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the voice patterns for the subjects scheduled were loaded 
into the machines (see Appendix C). The subjects were then 
Drought in and briefed and provided a three page handout. 
The last two pages were the same as for the practice 
scenario. Only the first page stating the situation, 
mission* and MOE's differed for each scenario (see Appendix 
E). The first page was the same for each position, except in 
one scenario when geographical coordinates were stated and 
were different for each subject. The briefing included an 
explanation of p ec u 1 iar i t i es within a given scenario which 
the subjects would not normally have experienced previously. 

Following the briefing, the actual NWISS session began. 
After the subjects went to their positions, the proper noise 
and lighting levels were set. and the game was started. All 
the sessions were run for a minimum of forty minutes. 
Personnel who had repeated difficulties were given 
assistance as necessary. At the conclusion of each run. 
copies of the game sessions were printed out and stamped for 
classification. The forty minute game time point was noted 
on the printouts, and the number of commands and “Control 
K '' s ,J entered were computed. 

F RESCHEDULING 

While the majority of the experiment sessions were 
completed as planned. some rescheduling was no-eded due to 
personnel being away on temporary duty status. or due to 
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Tie; cected NUilSS game problems Rescheduling of personnel 
c T ea tea no proolems because each input method was used to 
play an individual NWISS session. The proper scenario and 
snviTcnmental conditions could be set up for any individual 
makeup as required. 

£ SUMMARY 

This chapter has described the training of subjects in 
preparation for this experiment. All subjects trained with 
eitner the continuous or discrete mode of voice input. This 
was m order to give everyone an appreciation of the 
capabilities of voice input/ and to provide a sufficient 
case from which to assign students to an apparatus they had 
practiced on for the experiment. The chapter reviewed the 
practice sessions that took place* and the procedures used 
in conduct of the experiment 
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VI. 



DATA analysis 



A INTRODUCTION 

The objective of this experiment was to compare the 
effectiveness of three interactive computer input methods 
under several simulated command post environments. The data 
collected uias therefore analyzed with that in mind. The 
independent variables were two combinations of light and 
noise, and the dependent variables were the number of 
commands entered and the number of errors made. Each 
combination of environmental factors was analyzed from two 
aspects: first, comparing the number of commands entered as 
a positive measure of effectiveness; and second, examining 
the number of “Control K's" used as an indication of houj 
many errors were committed by the subjects. 

Tnis chapter will present summaries of the raw data, a 
trie* explanation of the statistical methods used, and 
report the results of the analysis in terms of the number of 
CiiX\i7\3T\d s entered and errors made. Through this analysis we 
hops to answer several questions. First, in which 
environment does each input method perform best based on the 
two factors of light and noise? Second, within each 
environment, were there significant differences among the 
input methods? And finally, if the environments are 
ignored, was there a "best' 1 input method for use in a 
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c o mma n d center. One additional question investigated that 
is not related to the environments is whether or not the 
subjects exhibited a learning curve as a result of multiple 
trials on the same input method. 

13 data summary 

Each command entered was a combination of NUilSS words 
needed to accomplish a desired task. Again, since the 
aircraft launch sequence consisted of five separate lines, 
eacn line correctly entered was counted as a command. The 
indication of an error was taken to be the use of the 
“Control ft 11 to abort a command for any reason. 

Tables 6. 1 and 6. 2 summarize the mean number of commands 
entered and errors for each input method by combination of 
environmental factors based on the data collected. Figures 
1 and 6.2 represent the same information in graphic form. 

TABLE 6. 1 

MEAN NUMBER OF COMMANDS ENTERED 
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Figure 6.2 Mean Number of "Control K 1 s " 
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TABLE fa. 2 



MEAN NUMBER OF “CONTROL K'S" 
Environment; Noise/Light 



Input: 
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Higures 6.3 and 6.4 summarize this data in graphic form. 
Raw aata is attached in Appendix F. 



TABLE fa. 3 



MEAN NUMBER OF COMMANDS BY TRIAL NUMBER 
Trial Number 



Input 
Me th od 
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Figure 6.4 Mean Number of "Control K ! s" 
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TABLE 6. 4 



MEAN NUMBER OF “CONTROL K'S" BY TRIAL NUMBER 

Trial Number 

Input 



Method 


i 


2 


3 


4 


’vertex 


ii. 15 


9. 54 


12. 69 


13. OS 


Threshold 


22. 70 


17. 00 


16. 50 


21. 40 


Keu board 


8. 08 


9. 00 


6. 08 


5. 15 



C STATISTICAL METHODS 

Analysis of variance techniques were used to compare 
performance in the four environments for each input method 
based on the two factors of light and noise; and to rank the 
methous within each environment. For most of the analyses; 
the sample size is thirteen; representative of the thirteen 
groups of three participants. 

T he two way analysis of variance procedure on the 
rtimtab application program of the IBM 3033 computer was 
used to determine what the environmental effects were on 
each piece of equipment and to find out if they perform 
Darter or worse in any particular environment of noise and 
light. The one way analysis of variance procedure (AOVO) 
which compares the means of several sets of data was used to 
compare the three input methods within each environment; 
without regard to the environment; and to determine if 
significant learning occurred through the trials. To use 
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these tests several assumptions must be made about the data 
Specifically; it is assumed that each set of data is a 
random sample,- each sample is normally distributed and each 
has approximately the same variance CRef 20: p. 1961. 

In all case Si the hypotheses were tested at a confidence 
level of 95 percent. If significant results were obtained at 
the 99 percent level* they are also reported. If the F 
ratio produced through the analysis of variance proved to be 
significant at the 95 percent level* the Newman-Keuls range 
test was applied to determine where the differences were for 
that particular set of data CRef 21: p. 353. 



U ANALYb'ib 

1 . Best Environment for Each Input Method 

Generally* we expected the voice input methods to do 
tetter in low light conditions and perhaps have less utility 
i n a high noise environment. We also anticipated that the 
keyboard would do worse in the low light environments. For 
the- purpose of analyzing the data* we hypothesized that the 
mean number of commands entered or the mean number of errors 
in each environment would be the same. Based on the F 
ratios calculated after two way analysis of variance at the 
95 percent confidence level* we were unable to reject these 
hypotheses for any of the three input methods. That is* 
none of the three input methods tested was significantly 
better or worse in all of the environments. 



BO 



Best Input Method for Each Environment 



This question is answered with the application of 
one wag analysis of variance testing with four sets of two 
similar null hypotheses. The two hypotheses were that 
either the mean number of commands entered or the mean 
number of errors committed were equal for the three input 
method*/ and they were compared in each of the four 
enviro n m snts. 

Based on the F ratio at a 95 percent confidence 
level, there is a significant difference in the mean number 
of errors made in the benign; normal light /low noise 
environment. In fact the F ratio was high enough to be 
significant at the 99 percent level. The Newman-Keu 1 s range 
test confirms that the mean number of errors produced by the 
keyboard and Vertex groups were not significantly different 
from each other even though the raw data showed the keyboard 
method with a slight advantage. Both were s i gn i f i can 1 1 y 
different from the Threshold score at a 99 percent 
confide n c e level. 

In the low noise/low light environment there were no 
statistically significant results. Observation of the data 
shows that Vertex and keyboard generally had higher numbers 
of commands entered and fewer errors than the Threshold 
method. Similar results occurred in the high noise/normal 
light environment. In the worst case; the high noise/ low 
light environment; the mean number of commands entered for 
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ail three methods were much closer together which again 
resulted in no significant differences. 

To summarize the results for the four environments^ 
only the benign environment produced s ta t i s t i ca 1 1 y 
significant results for the mean number of errors produced 
d y the three input methods. The Vertex and keyboard methods 
were essentially equal and Threshold significantly worse. 

3. Best Overall Input Method 

For these comparisons the effects of the environment 
were ignored/ and the average number of commands entered and 
i: Control K ' s " used from the four environments were tested 
against null hypotheses that the means were equal. Based on 
one way analysis of variance for the three methods/ at a 95 
percent conf idence level; these hypotheses can be rejected. 
Here.- too* the F ratios were significant at the 99 percent 
;e\ei. The alternative hypotheses that there were 
significant differences among both the mean number of 
commands entered and errors committed by the three groups 
are accepted. 

The Newman-Keu 1 s test revealed two significant 
results at the 95 percent confidence level. First; for the 
mean number of commands entered; Verbex and the keyboard 
were s t a t i s t i ca 1 1 y equal; and both were significantly higher 
than the Threshold method. Second; considering the mean 
number of “Control K's“ used; the manual entry method was 
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significantly lower than both of the voice entry methods; 
which aiere essentially equal. 

4 Learning Curves 

When considering the possiblity of a learning curve; 
the hypotheses were that no significant differences among 
the mean number of commands entered or "Control K's" used 
throughout the four trials. If there was a significant 
difference and the number of commands increased or the 

number of errors decreased; then significant learning took 
place. Although there were no significant differences 
detected based on the F ratios at a 95 percent confidence 
level; some learning did take place as graphs in Figures 6.3 
and 6.4 indicate. 

E. SUMMARY 

Briefly restating the results of the analysis of the 
data gathered in this experiment; there were few 

statistically significant results. None of the three input 
methods a t g o d out as being better or worse over all of the 
four combinations of light and noise. The keyboard and 

continuous speech recognition system did have noticeably 
fewer errors in the least stressful of the four 

environments. Without regard to the environment; the 

keyboard and Vertex groups were equal in their abilities to 

enter commands and able to enter more commands than the 

Threshold group. The keyboard method also produced fewer 
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errors than either of the voice entry techniques 
trie suDjects diu not exhibit a detectable learning 
over the course of four trials. 



Final 1 y , 
pattern 
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V I i . CONCLUb'IuNb ANu KtiCDMMENuAl lUNb 
A. INTRODUCTION 

Cased on the statistical analysis in Chapter VI, several 
conclusions can be drawn. These conclusions do not agree 
u;i tn our expectations in some cases. The purpose of this 
chapter is to explain some of the expected outcomes, their 
actual outcome and why they may have occurred. Also 
included in this chapter are implications for data entry 
methods in command and control centers, recommendations for 
future experiments, and applications of voice technologies. 

2 . CONCLUSIONS 

At the outset , we expected the voice input methods to 
i.. jtper f orrn the keyboard due to their more advanced 
technology. We also anticipated that the environments would 
have some significant effects on all three input methods. 
The statistics in the last chapter do not bear out these 
expected results. But the analysis of the data did confirm 
that the subjects did not learn how to use their input 
devices through the course of four trials. 

There were two reasons why the keyboard entry did as 
we]i as it did. First, most of the subjects wasted little 
time in moving their keyboards as close as possible to their 
data input screens in the low light condition using the 
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•video display's glow to assist them in finding the correct 
>eus Secondly; when the random draw was conducted to 

determine w h i c h subjects would use the keyboard; fate took a 
hand An informal survey of all subjects was conducted 

after the experiment to deter mine relative skill levels on 
the keyboard. The subjects were asked to rate themselves on 
the following scale: 

1/ Hunt and peck typist. 

(2: Able to use both hands; but must look at the keys. 

<3> "Expert" or touch typist. 

The results were that the Verbex group had a 1.69 average 
skill level; Threshold a 2.00 and keyboard a 2.54 average 
level. Keep in mind that this survey was purely subjective; 
cut it does help explain why the keyboard group faired so 

w ell. 

The experiment does not provide concrete statistical 
evidence that the keyboard is superior to voice 
technologies. The subjects in the keyboard group had 
rslativley superior typing skills; and yet the Verbex group 
proved to be statistically equal in their ability to enter 

i 

commands without regard to the ambient environment. None of 
the subjects in any of the groups could be considered as 
experts in the art of voice command interaction with a 
computer; while all had mors than a rudimentary knowledge of 
the use of a keyboard. 
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The background noise level also did not have a 
significant effect on the number of commands entered nor the 
number of errors. We noted three factors that may have 
influenced this outcome. First/ the high noise level 
provided very little distraction to the keyboard operator in 
particular/ and may have had the same minimal influence on 
the two voice inputs to a lesser degree. The keyboard 
operators found that they heard less of the Verbex and 
Threshold subjects'' commands when the higher noise level was 
used. 

This raises the second point: the background noise 
presented was essentially white noise/ with very few 
distinguishable words or phrases during the experiment. 
This may or may not be the case in a command center during a 

crisis. 

Finally/ the Verbex subjects may have experienced a 
difference between enviromnental conditions during training 
and the actual experimental environments/ which could have 
altered their performance. During training* access to the 
C3 lab was not restricted/ so people could come and go at 
will through the noisy/ air operated entry door. While the 
experiment was in progress/ access to the lab area was more 
tightly controlled so that the noise of the door and light 
in the foyer area would nGt affect the controlled interior 
environment. This should have resulted in fewer recognition 
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problems during testing# but there was no way of measuring a 

d inference. 

There is one area where our expectations were borne out 
Dy the data collected: the subjects did not experience a 

significant amount of learning during the course of the four 
trials. This result is desirable from the point of view 



t h a t 


all s 


ub j ec t s 


had reasonab 1 e 


f ami 1 iar i ty 


with the task 


t 0 


be p 


er f armed 


before the 


e x p er imen ta 1 


situation is 


pres 


e n t e d . 


We »:a n 


cone lude. then 


# that the 


comb inat ion of 


enro 


i 1 m e n t 


of the 


vocabularies/ 


three practi 


c e sessions and 



par t i c i pat i on in another NWISS related experiment adequately 
prepared the subjects for our experiment. 

C. IMPLICATIONS FOR DATA ENTRY METHODS 

Despite its age# the typewriter keyboard remains as a 
viable method to enter data or otherwise interact with a 
modern computer. It is impervious to the noise in its 

environment and is very nearly independent of the ambient 
lighting conditions provided its operator is well trained 
and has some means of feedback. The keyboard provides a 
high degree of accuracy though at times speed is sacrificed 
for accuracy. Finally# the keyboard is user independent# 
that is# anyone familiar with the QWERTY layout or other 
standard pattern can operate it with no additional training. 

Discrete voice recognition equipment# represented in 
this study by the Threshold T600# is really only one step 
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advanced from the keyboard. It accepts one utterance at a 
time versus one letter at a time. While this does represent 
a large step, it does not approach the current state of the 
art in voice input devices. Discrete voice equipment is 
also independent of the light in the environment# but it can 
be sensitive to the ambient noise. It does require some 
time on the part of the operator to train the device to 
recognize his or her voice. Without an adequate difference 
in the several samples given to the machine# a change in the 
stress level of the operator's voice can cause incorrect 
recognition. The use of voice recognition equipment does 
have a significant advantage over the keyboard in that the 
operator has the opportunity to perform some manual tasks in 
addition to interacting with the computer. This does 
present the possibility of overloading the operator# but the 
potential for performing manual tasks remains. 

Continuous voice recognition equipment such as the 
Vertex 3000 represents the state of the art in speech 
recognition technology. It offers greater potential for its 
users from the point of view that commands in the form of 
entire sentences can be employed. These can be recognized in 
considerably less time than is required to type them letter 
by letter on a keyboard or to enter them one word at a time 
as in a discrete voice system. But it is also operator 
dependent and therefore not as universal as a keyboard. 
Another shortcoming of the voice technology is the limited 
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metmory space 


available 


to store a number of 


possible 


entry 


sequences. 


The cont 


inuous 


voice system, 


though, 
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sensitive to 


the noise 


in the 


environment due 
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noise 


cance 1 lat i on 


feature. 


Like 


the Threshold 


system, 


Ver b e x 


offers the 


potential 


for 


limited manual 


tasks 


by the 



operator which could enhance his or her performance. 

D. RECOMMENDATIONS 

We offer these recommendations for further study of 
voice input and keyboard comparisons. The Threshold 
discrete recognition system performed significantly worse 
than Verbex in almost all cases. For further tests< pit the 
keyboard against the continuous recognition device in 
similar environments and ignore the Threshold system 
altogether. Regarding the environments# it may be necessary 
to use background noise with recognizable streams of words 
similar to NWISS commands to properly test the concentration 
of the subjects. In addition, the training environment 
should more closely resemble the experimental conditions if 
at all possible. If time permits, we also suggest that all 
subjects perform trials on both the voice equipment and the 
keyboard to reduce the possible advantage of the keyboard. 

We feel that NWISS should be the medium used for the 

/ 

experiment as it supports a large number of commands and is 
easily interfaced with the Vertex 3000 system. 
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SUrlhARY 



The advent of Very Large Scale Integration components in 
computer technology poses the possibility that memory 
devices can be made smaller than they are at present. This 
opens the door to more complex grammars in an automatic 
speech recognition system through increased memory capacity 
in less space. From the results of our experiment# although 
rhe keyboard turned in the best results for the type of task 
tested# we believe the future for continuous speech 
recognition systems is bright. Investigation into their 
integration in interactive computer programs for command and 
control center applications should be pursued. They offer 
the promise of tremendous benefits in applications where 
operators now using keyboard based devices are overwhelmed 
Dy the speed necessary to keep up with the flow of 
information. 
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APPENDIX A. 



VOCABULARY LISTS FOR EXPERIMENT 



THRESHOLD NWISS VOCABULARY 



PROMPT OUTPUT MEANING 



CANCEL 


(LINEFEED) 


TELL 


FOR 


CONSTELLATION 


CONNY 


PAUL 


PAUL 


ADAMS 


ADAMS 


CINCINATTI 


Cl NCI 


BARBIE 


BARBI 


ROANOKE 


ROANO 


GOLDSBOROUGH 


GOLDS 


MP7 


MP7 


SH7 


SH7 


VA7 


VA7 


VF7 


VF7 


VK7 


VK7 


VW7 


VW7 


1 POINT 2 


1. 2 


ZERO 


0 



CONTROL K 
ADDRESS FORCES 
SHIP NAME 
SHIP NAME 
SHIP NAME 
SUB NAME 
SHIP NAME 
SHIP NAME 
SHIP NAME 

P3C ASW ACFT CALL SIGNS 
SH2F ASW HELO CALL SIGNS 
A6E ATTACK ACFT CALL SIGNS 
F14A CAP ACFT CALL SIGNS 
KA6D AIRTANKER CALL SIGNS 
E2C EW ACFT CALL SIGNS 
FORCE COLLECTIVE CALL SIGN 
NUMBER 
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17 

iS 

19 

20 

21 

dc: 

23 

2A 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 



ONE 


1 


TWO 


2 


THREE 


3 


FOUR 


4 


FIVE 


5 


SIX 


6 


SEVEN 


7 


EIGHT 


8 


NINE 


9 


DESIGNATE 


DESIGNATE 


ENEMY 


ENEMY 


BEARING 


BEARING 


FORCE 


FORCE 


CENTER 


CENTER 


PLACE 


PLACE 


GRID 


GRID 


PLOT 


PLOT 


LOB 


LOB 


ERASE 


ERASE 


ESM 


ESM 


DROP 


DROP 


CANCEL 


CANCEL 


CIRCLE 


CIRCLE 


XMARK 


XMARK 


RADIUS 


RADIUS 


POSITION 


POSITION 



NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

CHANGE CONTACT TO ORANGE 
ORANGE 

DEGREES TRUE 
OWN UNIT 
CENTER OF PLOT 
PLACE GRAPHICS ON PLOT 
PLACE GRID ON PLOT 
PLOT SELECTED ELEMENTS 
ESM LINES OF BEARING 
ERASE ELEMENTS 
PLOT/CANCEL ESM LOBS 
ERASE TRACKS 
ERASE GRAPHIC ELEMENTS 
PLACE/CANCEL CIRCLE 
PLACE/CANCEL XMARK 
RADIUS OF PLOT 
LATITUDE/LONGITUDE 
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43 

44 

45 

46 

47 

48 

49 

50 

51 

«=,-> 

53 

54 

55 

56 

57 

58 

59 

60 

6 1 

62 

63 

64 

65 

66 

67 

oo 



- (TAC) 


— 


NORTH 


N 


SOUTH 


S 


EAST 


E 


WEST 


W 


TRACK 


TRACK 


AAO 


AAO 


AEO 


AEO 


APO 


APO 


ASO 


ASO 


AUO 


AUO 


FIRE 


FIRE 


CRUISE 


CRUISE 


HARPOON 


HRPON 


AT 


AT 


RANGE 


RANGE 


TORPEDO 


TORPEDO 


MARK 48 


MK48 


ASROC 


ASROC 


SPEED 


SPEED 


STATION 


STATION 


COURSE 


COURSE 


PROCEED 


PROCEED 


PERISCOPE 


PERISCOPE 


DEPTH 


DEPTH 


SURFACE 


SURFACE 



LAT/LONG EG 43-30(4 

DIRECTION 

DIRECTION 

DIRECTION 

DIRECTION 

ORANGE OR WHITE UNIT 
AIR TRACK 
ESM TRACK 

PASSIVE SONAR TRACK 
ACTIVE SURFACE TRACK 
ACTIVE SUBSURFACE TRACK 
FIRE CRUISE MIS/TORPS 
CRUISE MISSILE 
CRUISE MISSILE 
CRUISE MIS AT SHORE BASE 
DISTANCE IN NM 
ASW WEAPON 

SHIP/SUB FIRED TORPEDO 
SHIP/SUB FIRED TORPEDO 
VELOCITY IN KNOTS 
POSIT RELATIVE TO GUIDE 
HEADING IN DEGREES TRUE 
TRAVEL CUS/DIST OR POSIT 
SUB TO PERISCOPE DEPTH 
SUB DEPTH IN FEET 
SUB TO SURFACE 
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69 

70 

71 

72 

—J f\ 

74 

75 

76 

"7 "’ r 

7S 

79 

30 

81 

82 

83 

84 

85 

86 

87 

88 

89 

90 

91 

92 

93 

94 



MISSION 


MISSION 


SURCAP 


SURCAP 


SEARCH 


SEARCH 


CAP 


CAP 


STRIKE 


STRIKE 


AIRTANKER 


AIR TANKER 


STRIKE CAP 


STRCAP 


BINGO 


BINGO 


REFUEL 


REFUEL 


EMCON 


EMCON 


STOP 


STOP 


SILENT 


SILEN 


RADIATE 


RADIA 


TAKE 


TAKE 


COVER 


COVER 


ALTITUDE 


ALTITUDE 


RBOC 


RBOC 


ON 


ON 


DECM 


DECM 


OFF 


OFF 


BLIP 


BLIP 


WEAPONS 


WEAPONS 


TIGHT 


TIGHT 


FREE 


FREE 


ALL 


ALL 


AIR 


AIR 



MISSION OF UNIT 
SURV/CAP MISSION 
SEARCH MISSION 
COMBAT AIR PATROL MISSION 
STRIKE MISSION 
AIRTANKER MISSION 
STRIKECAP MISSION 
ACFT RETURN TO BASE 
CAUSE AIR REFUELING 
SELECT EMCON PLAN 
COMPLETE LAUNCH COMMAND 
EMCON SILENT (ALL OFF) 
EMCON RADIATE (ALL ON) 
SHOOT AT TRACK 
ACFT TRAILS TRACK 
ACFT ALTITUDE IN FEET 
USE SHIPBOARD CHAFF 
USE (RBOC/DECM/BLIP ) 

DEF ELEC COUNTER MEAS 
STOP USE (RBOC/DECM/BLIP) 
RADAR BLIP ENHANCER 
SET ROE (FREE/TIGHT) 

ROE = NO USE OF WEAPONS 
ROE = USE OF WEAPONS 
ENTIRE SET 
AIR SUBSET 
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95 

96 

97 

98 

99 

100 

101 

102 

103 

104 

105 

106 

107 

108 

109 

110 

111 

112 



launch 


LAUNCH 


A6E 


A6E 


E2C 


E2C 


F14A 


F14A 


KA6D 


KA6D 


P3C 


P3C 


SH2F 


SH2F 


LOAD 


LOAD 


MARK 83 


MK83 


PHOENIX 


PHENX 


SHRIKE 


SHRIK 


SPARROW 


SPAR 


SIDEWINDER 


SWDR 


WALLEYE 


WALL I 


SPACE 




SEND 


(CR) 


BACK-UP 


/(CHAR ) 


HELP 





LAUNCH ACFT COMMAND 
ATTACK ACFT 
EARLY WARNING ACFT 
FIGHTER/CAP ACFT 
AIR TANKER ACFT 
LAND BASED ASW ACFT 
ASW HELO 

LOAD ACFT W/ EXPENDIBLES 

AIR DROPPED BOMB 

AIR TO AIR MISSILE 

AIR TO SURFACE MISSILE 

AIR TO AIR MISSILE 

AIR TO AIR MISSILE 

AIR TO SURFACE MISSILE 

BLANK CHARACTER 

CARRIAGE RETURN 

ERASE CHARACTER (CTRL A) 

ASK FOR HELP 
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VERBEX NWISS VOCABULARY 



PROMPT SAY MEANING 



0 CONTROL 

1 FOR 

2 KITTY 

3 KNOX 

4 MCCOR 

5 OMAHA 

<b RATHB 

7 WICHI 

S WILSO 

9 MP6 

10 SHI 

i 1 VAO 

12 VFO 

1 3 VKO 

14 VWO 

15 1 POINT 

IS 0 

17 1 

18 2 

19 3 

20 4 

21 5 



K 



KITTYHAWK 

KNOX 

MCCORMICK 

OMAHA 

RATHBURN_ 

WICHITA 

WILSON 



CONTROL K 
ADDRESS FORCES 
SHIP NAME 
SHIP NAME 
SHIP NAME 
SUB NAME 
SHIP NAME 
SHIP NAME 
SHIP NAME 

P3C ASW ACFT CALL SIGNS 

SH2F ASW HELO CALL SIGNS 

A6E ATTACK ACFT CALL SIGNS 

F14A CAP ACFT CALL SIGNS 

KASD AIRTANKER CALL SIGNS 

E2C EW ACFT CALL SIGNS 

FORCE COLLECTIVE CALL SIGNS 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 

NUMBER 
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'D'D 

23 

24 

25 

26 

27 

2S 

29 

30 

31 

32 

33 

34 

35 

f-\ / 

oo 

•*3"T 

w / 

38 

39 

40 

41 

42 

43 

44 

45 

46 

f\ “7 

/ 



6 



NUMBER 



8 

9 

DESIGNATE 

ENEMY 

BEARING 

FORCE 

CENTER 

PLACE 

GRID 

PLOT 

LOB 

ERASE 

ESM 

DROP 

CANCEL 

CIRCLE 

XMARK 

RADIUS 

POSITION 

- (TAC) 

N 

S 

E 

W 



NORTH 

SOUTH 

EAST_ 

WEST 



NUMBER 

NUMBER 

NUMBER 

CHANGE CONTACT TO ORANGE 

ORANGE 

DEGREES TRUE 

OWN UNIT 

CENTER OF PLOT 

PLACE GRAPHICS ON PLOT 

PLACE GRID ON PLOT 

PLOT SELECTED ELEMENTS 

ESM LINES OF BEARING 

ERASE ELEMENTS 

PLOT/CANCEL ESM LOB 

ERASE TRACKS 

ERASE GRAPHIC ELEMENTS 

PLACE/CANCEL CIRCLE 

PLACE/CANCEL XMARK 

RADIUS OF PLOT 

LAT I TUDE/LONG I TUDE 

LAT/LONG EG 43-30N 

DIRECTION 

DIRECTION 

DIRECTION 

DIRECTION 
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48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 

61 

t> 2 

63 

64 

65 

66 

67 

63 

69 

70 

71 

72 

73 



TRACK 

BAO 

BEO 

BPO 

BSO 

EUO 

FIRE 

CRUISE 

HRPON 

AT 

RANGE 

TORPEDO 

MK48 

A3R0C 

SPEED 

STATION 

COURSE 

PROCEED 

PERISCOPE 

DEPTH 

SURFACE 

MISSION 

SURCAP 

SEARCH 

CAP 

STRIKE 



TRACK (ORANGE OR WHITE UNIT) 

AIR TRACK 

ESM TRACK 

PASSIVE SONAR TRACK 

ACTIVE SURFACE TRACK 

ACTIVE SUBSURFACE TRACK 

FIRE CRUISE MIS/TORPS 

CRUISE MISSILE 

HARPOON CRUISE MISSILE 

CRUISE MISSILE AT SHORE BASE 

DISTANCE IN NM 

ASW WEAPON 

SHIP/SUB FIRED TORPEDO 

SHIP/SUB FIRED TORPEDO 

VELOCITY IN KNOTS 

' POSIT RELATIVE TO GUIDE 

HEADING IN DEGREES TRUE 

TRAVEL COURSE OR POSIT 

SUB TO PERISCOPE DEPTH 

SUB DEPTH IN FEET 

SUB TO SURFACE 

MISSION OF UNIT 

SURV/CAP MISSION 

SEARCH MISSION 

COMBAT AIR PATROL MISSION 

STRIKE MISSION 
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/' 4 

75 

76 

77 

78 

79 

80 

81 

82 

83 

84 

35 

86 

87 

88 

89 

90 

91 

92 

93 

94 

95 

96 

97 

98 

99 



IRTANKER 



AIRTANKER MISSION 



STRCAP STRIKECAP MISSION 

EINGG ACFT RETURN TO BASE 

REFUEL CAUSE AIR REFUELING 

EMCON SELECT EMCON PLAN 

STOP COMPLETE LAUNCH COMMAND 



SILEN 


SILENT EMCON SILENT (ALL OFF) 


RADI A 


RADIATE EMCON RADIATE (ALL ON) 


TAKE 


SHOOT AT TRACK 


COVER 


ACFT TRAILS TRACK 


ALTITUDE 


ACFT ALTITUDE IN FEET 


RBOC 


USE SHIPBOARD CHAFF 


ON 


USE ( RBOC /DECM /BL IP ) 


DECM 


DEF ELEC COUNTER MEASURE 


GFF 


STOP USE ( RBOC /DECM/BLIP ) 


ELIP 


RADAR BLIP ENHANCER 


WEAPONS 


SET ROE (FREE/TIGHT) 


TIGHT 


ROE = NO USE OF WEAPONS 


FREE 


ROE = USE OF WEAPONS 


ALL 


ENTIRE SET 


AIR 


AIR SUBSET 


LAUNCH 


LAUNCH ACFT COMMAND 


A6E 


ATTACK ACFT 


E2C 


EARLY WARNING ACFT 


F14A 


FIGHTER/CAP ACFT 


KA6D 


AIRTANKER ACFT 



100 



100 

i 0 i 

102 

103 

104 

105 

iO£> 

107 

108 

109 

110 



F'3C 

SH2F 

LOAD 

MK83 




PHENX 


PHOENIX 


SHRIK 


SHRIK 


SPAR 


SPARROW 


SWDR 


SIDEWINDER 


WALL I 

DISPLAY 

VKOO 


WALLEYE 



LAND BASED ASW ACFT 
ASW HELO 

LOAD ACFT W/ EXPENDIBLES 
AIR DROPPED BOMB 
AIR TO AIR MISSILE 
AIR TO SURFACE MISSILE 
AIR TO AIR MISSILE 
AIR TO AIR MISSILE 
AIR TO SURFACE MISSILE 
ACCESS GRAPHICS 
CALL FOR REFUELING 
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0 

i 

2 

3 

4 

5 

6 

7 

8 

9 

i c 

1 1 

12 

A *~t 

14 

15 

16 

17 

IB 

19 

20 

21 



KEYBOARD NWISS VOCABULARY 



COMMAND 



CONTROL K 

FOR 

AMERI 

EAGLE 

TOWER 

MI SAW 

NYC 

FANNI 

WABAS 

STODD 

MPB 

SH8 

VAB 

VFB 

VK8 

vws 

1 POINT 3 
0 

1 

2 

3 

4 



MEANING 



CONTROL K 
ADDRESS FORCES 
SHIP NAME 
SHIP NAME 
SHIP NAME 
SHIP NAME 
SUB NAME 
SHIP NAME 
SHIP NAME 
SHIP NAME 

P3C ASW ACFT CALL SIGNS 
‘ SH2F ASW HELO CALL SIGNS 
A6E ATTACK ACFT CALL SIGNS 
F14A CAP ACFT CALL SIGNS 
KA6D AIRTANKER CALL SIGNS 
E2C EW ACFT CALL SIGNS 
FORCE COLLECTIVE CALL SIGN 
NUMBER 
NUMBER 
NUMBER 
NUMBER 
NUMBER 
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22 

23 

24 

z> ■=. 

26 

27 

26 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 



5 


NUMBER 


6 


NUMBER 


7 


NUMBER 


S 


NUMBER 


9 


NUMBER 


DESIGNATE 


CHANGE CONTACT DESIGNATION 


NEUTRAL 


WHITE 


FRIENDLY 


BLUE 


ENEMY 


ORANGE 


BEARING 


DEGREES TRUE 


FORCE 


OWN UNIT 


CENTER 


CENTER OF PLOT 


PLACE 


PLACE GRAPHICS ON PLOT 


GRID 


PLACE GRID ON PLOT 


PLOT 


PLOT SELECTED ELEMENTS 


LOB 


ESM LINES OF BEARING 


SONAR 


PLOT/ERASE SONAR LINES 


ERASE 


ERASE ELEMENTS 


e£m 


PLOT/CANCEL ESM LOB 


DROP 


ERASE TRACKS 


CANCEL 


ERASE GRAPHIC ELEMENTS 


CIRCLE 


PLACE/CANCEL CIRCLE 


XMARK 


PLACE/CANCEL XMARK 


RADIUS 


RADIUS OF PLOT 


POSITION 


LATITUDE/LONGITUDE 
LAT/LONG EG 43-30N 
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48 

49 

50 

51 

v-/ G. 

53 

54 

55 

58 

57 

58 

59 

80 

61 

62 

63 

£>4 

65 

66 

67 

68 

69 

70 

71 

72 

73 



N 


DIRECTION 


E 


DIRECTION 


W 


DIRECTION 


S 


DIRECTION 


TRACK 


TRACK (ORANGE OR WHITE UNIT) 


CAO 


AIR TRACK 


CEO 


ESM TRACK 


CPO 


PASSIVE SONAR TRACK 


CSO 


ACTIVE SURFACE TRACK 


CUO 


ACTIVE SUBSURFACE TRACK 


FIRE 


FIRE CRUISE MIS/TORPS 


CRUISE 


CRUISE MISSILE 


HRPON 


CRUISE MISSILE 


AT 


CRUISE MIS AT SHORE BASE 


RANGE 


DISTANCE IN NM 


TORPEDO 


ASW WEAPON 


MK48 


SHIP/SUB FIRED TORPEDO 


ASROC 


SHIP/SUB FIRED TORPEDO 


SPEED 


VELOCITY IN KNOTS 


STATION 


POSIT RELATIVE TO GUIDE 


COURSE 


HEADING IN DEGREES TRUE 


PROCEED 


TRAVELCUS/DIST OR POSIT 


PERISCOPE 


SUB TO PERISCOPE DEPTH 


DEPTH 


SUB DEPTH IN FEET 


SURFACE 


SUB TO SURFACE 


MISSION 


MISSION OF UNIT 
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74 

75 

76 

“7 “ 7 

/ / 

78 

79 

ao 

Si 

32 

S3 

84 

35 

86 

87 

88 

39 

90 

91 

92 

93 

94 

95 

96 

97 

98 

99 



SURCAP 


SURV/CAP MISSION 


SEARCH 


SEARCH MISSION 


CAP 


COMBAT AIR PATROL MISSION 


STRIKE 


STRIKE MISSION 


A I RT ANKER 


A I RT ANKER MISSION 


STRCAP 


STRIKECAP MISSION 


BINGO 


ACFT RETURN TO BASE 


REFUEL 


CAUSE AIR REFUELING 


EMC ON 


SELECT EMCON PLAN 


S I LEN 


EMCON SILENT (ALL OFF) 


RAD I A 


EMCON RADIATE (ALL ON) 


TAKE 


SHOOT AT TRACK 


COVER 


ACFT TRAILS TRACK 


ALTITUDE 


ACFT ALTITUDE IN FEET 


R30C 


USE SHIPBOARD CHAFF 


ON 


USE ( REOC/DECM/ELIP ) 


DECM 


DEF ELEC COUNTER MEAS 


OFF 


STOP USE (RBOC/DECM/BLIP ) 


BLIP 


RADAR BLIP ENHANCER 


WEAPONS 


SET ROE (FREE/TIGHT) 


TIGHT 


ROE = NO USE OF WEAPONS 


FREE 


ROE = USE OF WEAPONS 


ALL 


ENTIRE SET 


AIR 


AIR SUBSET 


LAUNCH 


LAUNCH ACFT COMMAND 


A6E 


ATTACK ACFT 
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100 

101 

102 

103 

104 

105 

106 

107 

ioe 

109 

110 

1 i 1 

112 
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A7E 


ATTACK ACFT 


E2C 


EARLY WARNING ACFT 


F14A 


FIGHTER/CAP ACFT 


KA6D 


AIRTANKER ACFT 


P3C 


LAND BASED ASW ACFT 


SH2F 


ASW HELO 


LOAD 


LOAD ACFT W/ EXPENDIBLES 


MK83 


AIR DROPPED BOMB 


PHENX 


AIR TO AIR MISSILE 


SHRIK 


AIR TO SURFACE MISSILE 


SPAR 


AIR TO AIR MISSILE 


SWDR 


AIR TO AIR MISSILE 


WALL I 


AIR TO SURFACE MISSILE 


STOP 


COMPLETE LAUNCH COMMAND 
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APPENDIX B. 



SCENARIO BRIEFINGS 
PRACTICE SCENARIO 

SITUATION: Relations with Orange have been deteriorating! 

and hostilities are expected to break out soon. An Orange 
SAG has been reported to the north. 

FRIENDLY FORCES: You have a carrier battle group with 

associated aircraft. 

ENEMY FORCES: Strength of Orange forces is unknown at this 

t ime. 

MISSION: You are to launch aircraft and search for and 

identify enemy forces. Weapons will remain tight unless 
fired upon; then you may respond in kind. 

MOE: POSITIVE 

-# of aircraft launched and loaded (HRPON's should be loaded 
on P3C ' s and A6E's> 

-# of enemy designated 

-# of enemy forces acquired/ hit/ and damaged 
NEGATIVE 

-# of enemy forces not designated 
-# of friendly forces damaged/lost 
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SCENARIO A 



SITUATION: Orange is escorting a large convoy of cargo ships 

containing nuclear and chemical weapons to one of its 
allies. Hostilities have been declared. 

FRIENDLY FORCES: You have 2 Knox class ships< a carrier with 

air assetsi and 2 flights of 4 P3C's available. The P3's are 
loaded with 4 Harpoon missiles and 4 MK46A torpedos. 
Add i tiona 1 ly< there is a Los Angeles class submarine whose 
position is north of the convoy. 

ENEMY FORCES: Orange has approx imately 8 ships escorting the 

convoy. There are approximately 10 merchant ships in the 
convoy. There are no enemy aircraft in the air< nor are they 
expected to launch any. 

MISSION: Destroy the Orange combatants using available 
assets without damaging any cargo vehicles (the type of 
cargo of the merchant ships makes any damage to them 
prohibitive). 

MOE: POSITIVE 

-# of weapons fired during play 

-# of combatants acquired; hit; and damaged/sunk 
NEGATIVE 

-# of merchants hit/sunk 
-# of Blue aircraft lost 

> 

-# of Blue vessals hit/sunk 
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SCENARIO B 



SITUATION: There are approximately 20 merchant ships at 

anchor waiting to unload at a port in one of Orange's newest 
colonies. Their cargo includes POL; spare aircraft parts, 
and ammunition to supply the militia of the colony. 
Diplomatic relations with Orange have been broken, and 
hostilities are imminent. 

FRIENDLY FORCES: Your forces include 2 P3C's loaded with 

Harpoons and MK46A torpedos, and two groups of 5 A6E's which 
are also loaded with Harpoons and torpedos. You also have a 
carrier available to launch additional forces. 

ENEMY FORCES: Orange is prepared to defend the merchant 

ships, but bad weather has not allowed us to verify its 
location or makeup. 

MISSION: You are to destroy as many merchant ships as 

possible, minimizing your own losses. If there are Orange 
forces in the area, you may attack if they fire on you. Your 
primary mission is the merchant ships. 

MOE: POSITIVE 

of weapons fired 

-# of merchant ships acquired, hit, and damaged 

of Orange combatants acquired, hit, and damaged 

NEGATIVE 

-# of Blue forces damaged/lost 
of merchant ships remaining 
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SCENARIO C 



SITUATION: Numerous Orange combatant elements have left 
their ports and are patrolling merchant ship lanes. They 
have been running EMCON silent and our national level 
sensors have been unable to locate them. There are friendly 
and neutral forces in the area also. Tensions are rising, 
but no hostilities have been declared. 

FRIENDLY FORCES: Blue has 1 P3 and 4 F14A's launched with 

standard loads. There are other Blue forces in the area, not 
under your control. Merchant ships are not hostile. 

ENEMY FORCES: There are enemy forces throughout the area. 

Enemy aircraft are not expected to be encountered. 

MISSION: Search for contacts in the area from 35N to 42N and 

from 174E to 1S0E. Locate and identify as many vessals as 
possible, and designate them as enemy, friendly, or neutral. 
Weapons will remain tight unless fired upon. Center your 
display upon your P3. Use only the f 1 i g h t/a ir craf t and any 
active track status boards. 

MOE: POSITIVE 

-# of enemy identified properly 

-# of friendly forces identified properly 

-# of neutral ships identified properly 

NEGATIVE 

-# of forces improperly designated 
— # of forces not found/designated 
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SCENARIO D 



SITUATION: Last night we received word that an Orange SAG 

was 200 NM north of our position/ and we have been steaming 
to intercept and visually identify their forces. At this 
time we should be ap p r o x ima t e 1 y 100 NM from them. 
Hostilities have not been declared/ but fighting is expected 
to break out soon. 

FRIENDLY FORCES: Blue has a carrier with associated 

aircraft/ 3 other ships/ and a submarine available. 

ENEMY FORCES: Strength of Orange forces is unknown. They are 

not expected to have aircraft launched at this time. 

MISSION: Launch carrier based assets to search for and 

identify possible Orange SAG. Weapons will remain tight 
unless fired upon. Then action is at your discretion. 

MOE: POSITIVE 

-# of aircraft launches 

of enemy assets identified 
-# of enemy assets damaged (if fired upon) 

NEGATIVE 

— # of Orange forces not designated as enemy 
-# of Elue aircraft lost 
-# of Blue ships hit/lost 
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APPENDIX C. 



VOICE EXPERIMENT TEAM MEMBERSHIP 



TEAM 


THRESHOLD 


KEYBOARD 


VERBEX 


1 


HENRY 


LARSON 


HUNZEKER 


2 


BAKER 


VONDERSCHEER 


VANE 


3 


TURNER 


DIASE 


HARDEE 


4 


BRENNAN 


SHELL 


WOLFRUM 


5 


CHAMBERLAIN 


SULLIVAN 


ANDERSON 


6 


SMART 


BASKEYFIELD 


LANDAY 


7 


KWIATKOWSKI 


YOUNG 


BOYD 


8 


BOURN 


JASKOT 


HAWRYLAK 


9 


COX 


ASHBY 


MILLER 


10 


CARLSON 


PETERSON 


GREENSPAN 


1 1 


CORBELL 


BERGER 


YEAKEL 


12 


AMBROSE 


JOHNSON 


BREIDERT 


13 


HUMPHRIES 


GATES 


TROY 



TEAM/SCENAR I Q SCHEDULE 



DAY 


TIME 


TIME 


TIME 


TIME 


TIME 


TIME 


TIME 


TIME TIME 






12-13 


13-14 


14-15 


08-09 


09-10 


10-1 1 


14-15 


15-16 


16-17 


30 


OCT 


IP! 


*■* 


2P1 














31 


OCT 








3P1 


-*■* 


4P1 


5P1 


6P1 


7P1 


1 


NOV 














8P1 


9P1 




2 


NOV 








10P1 


1 IP! 


## 


12P 1 


13P 1 




6 


NOV 


1A1 




2B2 














7 


NOV 








3C3 




4B1 


5D3 


6A4 


7C1 


8 


NOV 














BD2 


9B4 


8C1 


9 


NOV 








10D1 


11A2 


1 0C4 


12B3 


13C4 


12A2 


13 


NOV 


1D4 


** 


2A1 














14 


NOV 








3B2 


*«■# 


4A4 


5C2 


6D3 


7B4 


15 


NOV 














BA3 


9A3 


9C1 


16 


NOV 








10A2 


11D1 


1 1 B3 


1 2C4 


13B3 


13D1 


20 


NOV 


1B2 


** 


2C3 














21 


NOV 








3D4 




4C2 


5A4 


6B1 


7D2 


23 


NOV 


TEAM 


RECAP 


DAY 














27 


NOV 


1C3 




2D4 














28 


NOV 








3A1 


■*# 


4D3 


5B1 


6C2 


7A3 


29 


NOV 










i 




8B4 


9D2 


-*# 


30 


NOV 








1 0B3 


ifc4 




12D1 


1 3A2 
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APPENDIX D. 



ADMINISTRATIVE BRIEFING 

- Subjects were to wait outside the lab until told to come 
in. This would prevent disturbing the lab environment if a 
group was scheduled before them. 

- Sessions would last ap pr o x ima te 1 y fifty minutes. 

- Voice patterns were already loaded for the voice users. 
Vertex users simply had to set their gain to be ready to 
start. 

- The noise and light conditions for the exercise were 

explained. , 

- The set-ups were explained. 

- Subjects were advised to ask for help during the practice 
session if they ran into problems. 

- If voice subjects still had trouble with inputs after 
three a t temp t s» they were advised to revert to keyboard 
entry to finish that command. They were reminded that they 
could retrain utterances any time there was not an NWISS 
session in progress 

- Subjects were asked not to discuss any scenarios following 
their sessions until the experiment was completed. 

- The group was reminded of their next scheduled lab period 
for the experiment. 
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APPENDIX E. 



SAMPLE TASKS TO BE PERFORMED 

-fire narpoon cruise missiles from aircraft or ship 
-fire torpedos at tracks 
-change depth of submarine 

-change speed, course, or position of blue forces 

-determine bearing and range from friendly force to desired track 
-go weapons free on enemy or all 
-change emcon status 
-launch and load aircraft 

-place circles of desired radius around a force or position 
-designate tracks as enemy, friendly, or neutral 
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APPENDIX F. 



INDIVIDUAL TEST SESSION DATA 











TOTAL 


COMMANDS 


TOTAL 


CONTROL K's 


GRP 


SCN 


ENV 


ORD 


VER 


THR 


KEY 


VER 


THR 


KEY 


A 

L 


A 


i 


1 


26 


65 


50 


11 


3 


2 




E 


2 


3 


65 


67 


69 


20 


3 


5 




C 


3 


4 


40 


50 


55 


5 


6 


1 




D 


4 


c- 


51 


66 


62 


20 


3 


3 




A 


1 


2 


67 


45 


64 


9 


1 1 


13 




B 


2 


1 


56 


46 


4i 


10 


1 1 


4 




c 


3 


3 


49 


37 


63 


4 


13 


1 1 




D 


4 


4 


57 


61 


65 


7 


5 


2 


O 

u 


A 


1 


4 


80 


28 


85 


13 


24 


9 




B 


rj 

CL 


2 


81 


38 


59 


12 


17 


33 
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1 


47 


20 


36 


8 


58 
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D 
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59 


61 


65 


8 


24 


16 


A 


A 


4 


2 


85 


75 
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14 


8 


12 




B 


i 


1 


70 


51 


98 


13 


20 


14 




C 


2 


3 


81 


46 


89 


7 


6 


5 




D 


3 


4 


81 


73 


126 


i2 


10 


10 


S 


A 


4 


3 
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43 


57 


15 


5 


2 




B 


1 


4 
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52 


61 


10 


42 


0 




C 


n 

e. 


n 


69 


54 


34 


1 


1 


1 




D 


3 


1 


68 


44 


36 


13 


5 


8 


6 


A 


4 


1 
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76 


98 


19 


46 


23 
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1 


3 


138 


75 


88 


22 


34 


2 




C 


2 


4 


e7 


68 


77 


17 


55 


10 




D 


3 


2 


94 


78 


98 


19 


34 


15 


7 


A 


3 


4 


77 


47 


115 


10 


32 
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B 


4 


2 


50 


44 


86 


8 


29 


4 




C 


1 


1 


35 


44 


52 


6 


19 
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D 


2 


3 


60. 


30 
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2 


8 


6 
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A 
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3 


81 


40 


63 


28 


i9 
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4 


4 


76 


45 


82 


15 


22 
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C 


1 


2 


37 


51 


70 


3 


20 


7 




D 


2 


1 


67 


38 


62 


8 


18 


7 
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? 



A 3 2 

E 4 1 

C 1 3 

D 2 4 

10 A 2 3 

E 3 4 

C 4 2 

D 1 1 

11 A 2 1 

E 3 3 

C 4 4 

D 1 2 

12 A 2 2 

B 3 1 

C 4 3 

D 1 4 

13 A 2 4 

B 3 2 

C 4 1 

D 1 3 



69 


67 


88 


4 


17 


4 


79 


58 


65 


5 


38 


6 


60 


60 


61 


5 


20 


4 


65 


89 


78 


11 


17 


2 
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89 


78 


28 


24 


6 


96 


98 


79 


40 


15 


5 


74 


57 


68 


3 


30 


5 


95 


73 


88 


13 


12 


3 


60 


34 


64 


26 


19 


10 


59 


39 


98 


13 


34 


8 


57 


37 


73 


14 


25 


5 


65 


54 


80 


10 


16 


7 


52 


50 


89 


14 


17 


6 


56 


51 


92 


6 


30 


6 


45 


79 


48 


3 


10 


1 


48 


79 


94 


6 


12 


4 


58 


85 


92 


10 


13 


5 


54 


77 


78 


7 


18 


6 


30 


62 


46 


7 


16 


7 


75 


93 


72 


10 


14 


3 



GRP=GR0UP 

SCN=SCENaRI0 

0RD=0RDER OF OCCURANCE 

VER=VERBEX 

THR=THRESHOLD 

KEY=KEYBOARD 
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